2024-01-15T14:24:04.899 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-15T14:24:04.899 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082 --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --name rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi --verbose -- /home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082/orig.config.yaml 2024-01-15T14:24:04.941 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T14:24:05.026 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7517082' kernel: client: sha1: distro flavor: default kdb: true sha1: c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-vol-clone-stats-temp sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2171 sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b sleep_before_teardown: 0 suite: fs:volumes suite_branch: mgr-vol-clone-stats-temp suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAm4BXul+XHUCskQiKefJarpQ9/RKYyRFJFCclOpgH06y89Jd2ZojDz9DQif+pbcTJGbWq9qELPAYP9m+tbmoHw= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSr5y6GfvbWAbi7CR/OsML6+UDb2zXubrvEUEnenhEBJeiHQSGr42UrAhK7RFT/ydM05oOJJLic2pNGBqryQ5g= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_14:18:41 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-15T14:24:05.026 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-15T14:24:05.026 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-15T14:24:05.026 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-15T14:24:05.028 INFO:teuthology.task.internal:Checking packages... 2024-01-15T14:24:05.052 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '95c90f413c4bb65e3e56909855f2308fd9bda05b' 2024-01-15T14:24:05.052 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-15T14:24:05.052 INFO:teuthology.packaging:ref: None 2024-01-15T14:24:05.052 INFO:teuthology.packaging:tag: None 2024-01-15T14:24:05.052 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-15T14:24:05.052 INFO:teuthology.packaging:sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T14:24:05.052 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-clone-stats 2024-01-15T14:24:05.278 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g95c90f41 2024-01-15T14:24:05.279 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-15T14:24:05.289 INFO:teuthology.task.internal:no buildpackages task found 2024-01-15T14:24:05.289 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-15T14:24:05.298 INFO:teuthology.task.internal:Saving configuration 2024-01-15T14:24:05.312 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-15T14:24:05.318 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T14:24:05.341 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 14:18:50.887352', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAm4BXul+XHUCskQiKefJarpQ9/RKYyRFJFCclOpgH06y89Jd2ZojDz9DQif+pbcTJGbWq9qELPAYP9m+tbmoHw='} 2024-01-15T14:24:05.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 14:18:50.885543', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSr5y6GfvbWAbi7CR/OsML6+UDb2zXubrvEUEnenhEBJeiHQSGr42UrAhK7RFT/ydM05oOJJLic2pNGBqryQ5g='} 2024-01-15T14:24:05.360 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-15T14:24:05.367 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-15T14:24:05.367 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-15T14:24:05.367 INFO:teuthology.run_tasks:Running task console_log... 2024-01-15T14:24:05.438 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f54036a6160>, signals=[15]) 2024-01-15T14:24:05.439 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-15T14:24:05.445 INFO:teuthology.task.internal:Opening connections... 2024-01-15T14:24:05.445 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:24:05.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:05.517 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:24:05.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:24:05.591 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-15T14:24:05.598 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-01-15T14:24:05.618 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-01-15T14:24:05.618 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Red Hat Enterprise Linux" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8.6 (Ootpa)" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:ID="rhel" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="fedora" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8.6" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.redhat.com/" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-15T14:24:05.676 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T14:24:05.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:05.677 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-15T14:24:05.677 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-15T14:24:05.677 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-15T14:24:05.677 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-15T14:24:05.677 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-01-15T14:24:05.701 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-01-15T14:24:05.721 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-01-15T14:24:05.721 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-01-15T14:24:05.780 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Red Hat Enterprise Linux" 2024-01-15T14:24:05.780 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8.6 (Ootpa)" 2024-01-15T14:24:05.780 INFO:teuthology.orchestra.run.smithi159.stdout:ID="rhel" 2024-01-15T14:24:05.780 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="fedora" 2024-01-15T14:24:05.780 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8.6" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.redhat.com/" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-15T14:24:05.781 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-15T14:24:05.781 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-01-15T14:24:05.800 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-15T14:24:05.809 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-15T14:24:05.815 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-15T14:24:05.816 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-01-15T14:24:05.818 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-01-15T14:24:05.837 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-15T14:24:05.843 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-15T14:24:05.843 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-01-15T14:24:05.877 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-01-15T14:24:05.903 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-15T14:24:05.958 INFO:teuthology.run_tasks:Running task kernel... 2024-01-15T14:24:05.971 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b'} 2024-01-15T14:24:05.971 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-01-15T14:24:05.971 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-01-15T14:24:05.972 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T14:24:05.972 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T14:24:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:24:05.991 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-01-15T14:24:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:24:05.991 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-01-15T14:24:06.050 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:06.050 INFO:teuthology.task.kernel:Running kernel on smithi159: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:06.050 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-01-15T14:24:06.051 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:06.051 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:06.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-01-15T14:24:07.613 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2024-01-15T14:24:07.613 INFO:teuthology.orchestra.run.smithi159.stdout:Unable to read consumer identity 2024-01-15T14:24:07.647 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-15T14:24:07.647 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2024-01-15T14:24:07.743 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:07.743 INFO:teuthology.orchestra.run.smithi159.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T14:24:07.743 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:07.786 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:07.786 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T14:24:07.786 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:09.623 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2024-01-15T14:24:09.666 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-01-15T14:24:10.776 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-01-15T14:24:11.003 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-01-15T14:24:14.382 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 305 kB/s | 24 kB 00:00 2024-01-15T14:24:14.801 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 164 kB/s | 24 kB 00:00 2024-01-15T14:24:15.118 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-15T14:24:15.118 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-15T14:24:15.201 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-15T14:24:15.202 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-01-15T14:24:15.202 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-15T14:24:15.339 DEBUG:teuthology.orchestra.run.smithi077:> echo no | sudo yum reinstall kernel || true 2024-01-15T14:24:15.536 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-15T14:24:15.536 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-15T14:24:15.602 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-01-15T14:24:15.603 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-01-15T14:24:15.603 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-01-15T14:24:15.713 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-15T14:24:15.713 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2024-01-15T14:24:15.766 DEBUG:teuthology.orchestra.run.smithi159:> echo no | sudo yum reinstall kernel || true 2024-01-15T14:24:15.771 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:15.771 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T14:24:15.771 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:16.082 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 02:24:14 PM UTC. 2024-01-15T14:24:16.105 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2024-01-15T14:24:16.105 INFO:teuthology.orchestra.run.smithi159.stdout:Unable to read consumer identity 2024-01-15T14:24:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:16.162 INFO:teuthology.orchestra.run.smithi159.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T14:24:16.163 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:16.174 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T14:24:16.174 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T14:24:16.210 INFO:teuthology.orchestra.run.smithi077.stderr:Error: No packages marked for reinstall. 2024-01-15T14:24:16.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum reinstall -y kernel || true 2024-01-15T14:24:16.478 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 02:24:14 PM UTC. 2024-01-15T14:24:16.548 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T14:24:16.563 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T14:24:16.588 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-15T14:24:16.589 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2024-01-15T14:24:16.617 INFO:teuthology.orchestra.run.smithi159.stderr:Error: No packages marked for reinstall. 2024-01-15T14:24:16.645 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:16.645 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T14:24:16.646 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:24:16.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum reinstall -y kernel || true 2024-01-15T14:24:16.951 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 02:24:14 PM UTC. 2024-01-15T14:24:17.001 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2024-01-15T14:24:17.002 INFO:teuthology.orchestra.run.smithi159.stdout:Unable to read consumer identity 2024-01-15T14:24:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T14:24:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T14:24:17.050 INFO:teuthology.orchestra.run.smithi077.stderr:Error: No packages marked for reinstall. 2024-01-15T14:24:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:17.054 INFO:teuthology.orchestra.run.smithi159.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T14:24:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:24:17.098 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-01-15T14:24:17.265 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:17.265 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:17.265 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-15T14:24:17.265 DEBUG:teuthology.parallel:result is None 2024-01-15T14:24:17.363 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:03 ago on Mon 15 Jan 2024 02:24:14 PM UTC. 2024-01-15T14:24:17.425 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T14:24:17.425 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T14:24:17.464 INFO:teuthology.orchestra.run.smithi159.stderr:Error: No packages marked for reinstall. 2024-01-15T14:24:17.518 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-01-15T14:24:17.684 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:17.685 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T14:24:17.685 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-15T14:24:17.685 DEBUG:teuthology.parallel:result is None 2024-01-15T14:24:17.685 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-15T14:24:17.698 INFO:teuthology.task.internal:Creating test directory... 2024-01-15T14:24:17.699 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T14:24:17.701 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T14:24:17.719 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-15T14:24:17.726 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-15T14:24:17.734 INFO:teuthology.task.internal:Creating archive directory... 2024-01-15T14:24:17.734 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T14:24:17.760 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T14:24:17.798 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-15T14:24:17.807 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-15T14:24:17.808 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T14:24:17.827 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T14:24:17.856 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T14:24:17.872 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T14:24:17.873 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T14:24:17.892 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T14:24:17.894 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-15T14:24:17.901 INFO:teuthology.task.internal:Configuring sudo... 2024-01-15T14:24:17.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T14:24:17.917 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T14:24:17.961 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-15T14:24:17.970 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-15T14:24:17.970 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T14:24:17.985 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T14:24:18.018 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T14:24:18.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T14:24:18.134 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T14:24:18.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T14:24:18.260 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:24:18.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T14:24:18.327 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T14:24:18.371 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T14:24:18.422 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T14:24:18.504 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T14:24:18.547 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:24:18.548 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T14:24:18.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-01-15T14:24:18.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-01-15T14:24:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T14:24:18.702 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T14:24:19.092 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-15T14:24:19.100 INFO:teuthology.task.internal:Starting timer... 2024-01-15T14:24:19.101 INFO:teuthology.run_tasks:Running task pcp... 2024-01-15T14:24:19.188 INFO:teuthology.run_tasks:Running task selinux... 2024-01-15T14:24:19.197 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-15T14:24:19.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-01-15T14:24:19.269 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-01-15T14:24:19.271 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-01-15T14:24:19.361 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-01-15T14:24:19.363 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-15T14:24:19.363 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-01-15T14:24:19.399 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-01-15T14:24:19.399 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-01-15T14:24:19.436 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-01-15T14:24:19.436 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-01-15T14:24:19.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T14:24:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:24:19.468 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T14:24:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:24:19.504 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-15T14:24:19.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-01-15T14:24:19.537 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-01-15T14:24:19.580 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-15T14:24:19.591 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-15T14:24:19.597 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-15T14:24:19.757 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-15T14:24:19.771 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-15T14:24:19.772 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-15T14:31:18.340 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-01-15T14:31:18.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T14:31:18.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:31:18.414 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-01-15T14:31:18.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T14:31:18.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T14:31:18.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T14:31:18.556 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-01-15T14:31:18.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T14:31:18.630 INFO:teuthology.run_tasks:Running task clock... 2024-01-15T14:31:18.642 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-15T14:31:18.642 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T14:31:18.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T14:31:18.645 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T14:31:18.645 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T14:31:18.690 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T14:31:18.703 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T14:31:18.718 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T14:31:18.763 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-01-15T14:31:18.774 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-01-15T14:31:18.788 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-01-15T14:31:18.789 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T14:31:18.801 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-01-15T14:31:18.803 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.905 INFO:teuthology.orchestra.run.smithi159.stderr:bash: ntpq: command not found 2024-01-15T14:31:18.909 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T14:31:18.909 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-01-15T14:31:18.909 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.909 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.910 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.910 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T14:31:18.910 INFO:teuthology.run_tasks:Running task install... 2024-01-15T14:31:18.922 DEBUG:teuthology.task.install:project ceph 2024-01-15T14:31:18.922 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}} 2024-01-15T14:31:18.922 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'} 2024-01-15T14:31:18.922 INFO:teuthology.task.install:Using flavor: default 2024-01-15T14:31:18.927 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-15T14:31:18.927 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-15T14:31:18.927 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'tag': None, 'wait_for_package': False} 2024-01-15T14:31:18.928 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T14:31:18.929 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'tag': None, 'wait_for_package': False} 2024-01-15T14:31:18.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T14:31:19.148 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/ 2024-01-15T14:31:19.148 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g95c90f41 2024-01-15T14:31:19.176 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/ 2024-01-15T14:31:19.177 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g95c90f41 2024-01-15T14:31:19.272 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-15T14:31:19.272 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:31:19.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-15T14:31:19.301 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-15T14:31:19.301 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/95c90f413c4bb65e3e56909855f2308fd9bda05b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-15T14:31:19.359 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-15T14:31:19.359 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:31:19.359 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-15T14:31:19.369 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-15T14:31:19.388 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-15T14:31:19.388 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/95c90f413c4bb65e3e56909855f2308fd9bda05b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-15T14:31:19.446 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-15T14:31:19.457 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-01-15T14:31:19.514 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-01-15T14:31:19.535 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-15T14:31:19.602 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-01-15T14:31:19.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-01-15T14:31:19.890 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-15T14:31:19.981 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2024-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi077.stdout:57 files removed 2024-01-15T14:31:20.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi159.stdout:57 files removed 2024-01-15T14:31:20.332 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-15T14:31:20.614 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-15T14:31:20.668 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2024-01-15T14:31:21.392 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 295 kB/s | 81 kB 00:00 2024-01-15T14:31:21.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 187 kB/s | 81 kB 00:00 2024-01-15T14:31:21.760 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 29 kB/s | 9.5 kB 00:00 2024-01-15T14:31:21.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 30 kB/s | 9.5 kB 00:00 2024-01-15T14:31:22.177 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 3.8 kB/s | 1.5 kB 00:00 2024-01-15T14:31:22.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 3.8 kB/s | 1.5 kB 00:00 2024-01-15T14:31:22.352 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-01-15T14:31:22.406 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-15T14:31:23.147 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2024-01-15T14:31:26.758 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 287 kB/s | 24 kB 00:00 2024-01-15T14:31:27.180 INFO:teuthology.orchestra.run.smithi159.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 25 MB/s | 8.8 MB 00:00 2024-01-15T14:31:27.665 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 409 kB/s | 24 kB 00:00 2024-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 65 MB 00:01 2024-01-15T14:31:29.723 INFO:teuthology.orchestra.run.smithi159.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 58 MB 00:01 2024-01-15T14:31:39.076 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 63 MB/s | 58 MB 00:00 2024-01-15T14:31:40.832 INFO:teuthology.orchestra.run.smithi159.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 65 MB 00:01 2024-01-15T14:31:49.435 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 28 MB/s | 8.8 MB 00:00 2024-01-15T14:31:55.401 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T14:31:55.403 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T14:31:55.421 INFO:teuthology.orchestra.run.smithi159.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-15T14:31:55.423 INFO:teuthology.orchestra.run.smithi159.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-15T14:31:55.424 INFO:teuthology.orchestra.run.smithi159.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-15T14:31:55.426 INFO:teuthology.orchestra.run.smithi159.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-15T14:31:55.429 INFO:teuthology.orchestra.run.smithi159.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-15T14:31:55.433 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-15T14:31:55.434 INFO:teuthology.orchestra.run.smithi159.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-15T14:31:55.638 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-01-15T14:31:55.652 INFO:teuthology.orchestra.run.smithi159.stdout:=================================================================================================================================== 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout:=================================================================================================================================== 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.4 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 5.4 M 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 920 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 171 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 1.6 M 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 133 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 1.8 M 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 7.4 M 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 50 k 2024-01-15T14:31:55.653 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 14 M 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 49 M 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 262 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 412 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 263 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 18 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 31 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 839 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 139 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-15T14:31:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 197 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 370 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 364 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 107 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 94 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.6 M 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 181 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-15T14:31:55.655 INFO:teuthology.orchestra.run.smithi159.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.8 M 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.7 M 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-01-15T14:31:55.656 INFO:teuthology.orchestra.run.smithi159.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 23 M 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 28 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 2.3 M 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 246 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 4.6 M 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 18 M 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 15 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 24 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 183 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-15T14:31:55.657 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 517 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.9 M 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.3.4-12.el8 codeready-builder-for-rhel-8-x86_64-rpms 30 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-01-15T14:31:55.658 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 45 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 128 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-15T14:31:55.659 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-15T14:31:55.660 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout:Enabling module streams: 2024-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql 10 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout:=================================================================================================================================== 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout:Install 131 Packages 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 16 Packages 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 218 M 2024-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-01-15T14:31:56.140 INFO:teuthology.orchestra.run.smithi159.stdout:(1/147): ceph-18.0.0-7629.g95c90f41.el8.x86_64. 21 kB/s | 6.4 kB 00:00 2024-01-15T14:31:56.374 INFO:teuthology.orchestra.run.smithi159.stdout:(2/147): ceph-base-18.0.0-7629.g95c90f41.el8.x8 9.8 MB/s | 5.4 MB 00:00 2024-01-15T14:31:56.463 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T14:31:56.464 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-15T14:31:56.483 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-15T14:31:56.486 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-15T14:31:56.488 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-15T14:31:56.490 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-15T14:31:56.495 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-15T14:31:56.496 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-15T14:31:56.540 INFO:teuthology.orchestra.run.smithi159.stdout:(3/147): ceph-immutable-object-cache-18.0.0-762 1.0 MB/s | 171 kB 00:00 2024-01-15T14:31:56.700 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-15T14:31:56.714 INFO:teuthology.orchestra.run.smithi077.stdout:=================================================================================================================================== 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout:=================================================================================================================================== 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.4 k 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 5.4 M 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 920 k 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 171 k 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 1.6 M 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 133 k 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 1.8 M 2024-01-15T14:31:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 7.4 M 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 50 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 14 M 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 49 M 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 262 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 412 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 263 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 18 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 31 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 839 k 2024-01-15T14:31:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 139 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 197 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 370 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 364 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 107 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 94 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.6 M 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 181 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-15T14:31:56.717 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.8 M 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 3.7 M 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-15T14:31:56.718 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 23 M 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 28 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 2.3 M 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 246 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 4.6 M 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 18 M 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g95c90f41.el8 ceph-noarch 15 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 24 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-15T14:31:56.719 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 183 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 517 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 6.9 M 2024-01-15T14:31:56.720 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.3.4-12.el8 codeready-builder-for-rhel-8-x86_64-rpms 30 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-15T14:31:56.721 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 45 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g95c90f41.el8 ceph 128 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-15T14:31:56.722 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-15T14:31:56.723 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-15T14:31:56.724 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql 10 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout:=================================================================================================================================== 2024-01-15T14:31:56.725 INFO:teuthology.orchestra.run.smithi077.stdout:Install 131 Packages 2024-01-15T14:31:56.726 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 16 Packages 2024-01-15T14:31:56.726 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:31:56.726 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 218 M 2024-01-15T14:31:56.726 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-01-15T14:31:56.791 INFO:teuthology.orchestra.run.smithi159.stdout:(4/147): ceph-mds-18.0.0-7629.g95c90f41.el8.x86 9.3 MB/s | 2.3 MB 00:00 2024-01-15T14:31:57.025 INFO:teuthology.orchestra.run.smithi159.stdout:(5/147): ceph-mgr-18.0.0-7629.g95c90f41.el8.x86 7.0 MB/s | 1.6 MB 00:00 2024-01-15T14:31:57.102 INFO:teuthology.orchestra.run.smithi077.stdout:(1/147): ceph-18.0.0-7629.g95c90f41.el8.x86_64. 19 kB/s | 6.4 kB 00:00 2024-01-15T14:31:57.527 INFO:teuthology.orchestra.run.smithi159.stdout:(6/147): ceph-mon-18.0.0-7629.g95c90f41.el8.x86 9.3 MB/s | 4.6 MB 00:00 2024-01-15T14:31:57.758 INFO:teuthology.orchestra.run.smithi159.stdout:(7/147): ceph-common-18.0.0-7629.g95c90f41.el8. 12 MB/s | 23 MB 00:01 2024-01-15T14:31:58.000 INFO:teuthology.orchestra.run.smithi159.stdout:(8/147): ceph-fuse-18.0.0-7629.g95c90f41.el8.x8 495 kB/s | 920 kB 00:01 2024-01-15T14:31:58.144 INFO:teuthology.orchestra.run.smithi077.stdout:(2/147): ceph-fuse-18.0.0-7629.g95c90f41.el8.x8 884 kB/s | 920 kB 00:01 2024-01-15T14:31:58.361 INFO:teuthology.orchestra.run.smithi077.stdout:(3/147): ceph-immutable-object-cache-18.0.0-762 789 kB/s | 171 kB 00:00 2024-01-15T14:31:58.575 INFO:teuthology.orchestra.run.smithi159.stdout:(9/147): ceph-selinux-18.0.0-7629.g95c90f41.el8 42 kB/s | 24 kB 00:00 2024-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi159.stdout:(10/147): ceph-osd-18.0.0-7629.g95c90f41.el8.x8 9.9 MB/s | 18 MB 00:01 2024-01-15T14:31:59.484 INFO:teuthology.orchestra.run.smithi159.stdout:(11/147): ceph-radosgw-18.0.0-7629.g95c90f41.el 8.0 MB/s | 14 MB 00:01 2024-01-15T14:31:59.509 INFO:teuthology.orchestra.run.smithi159.stdout:(12/147): cephfs-mirror-18.0.0-7629.g95c90f41.e 1.6 MB/s | 263 kB 00:00 2024-01-15T14:31:59.519 INFO:teuthology.orchestra.run.smithi077.stdout:(4/147): ceph-base-18.0.0-7629.g95c90f41.el8.x8 1.9 MB/s | 5.4 MB 00:02 2024-01-15T14:31:59.526 INFO:teuthology.orchestra.run.smithi159.stdout:(13/147): libcephfs-devel-18.0.0-7629.g95c90f41 747 kB/s | 31 kB 00:00 2024-01-15T14:31:59.643 INFO:teuthology.orchestra.run.smithi159.stdout:(14/147): libcephsqlite-18.0.0-7629.g95c90f41.e 1.5 MB/s | 183 kB 00:00 2024-01-15T14:31:59.677 INFO:teuthology.orchestra.run.smithi159.stdout:(15/147): libcephfs2-18.0.0-7629.g95c90f41.el8. 4.9 MB/s | 839 kB 00:00 2024-01-15T14:31:59.711 INFO:teuthology.orchestra.run.smithi159.stdout:(16/147): librados-devel-18.0.0-7629.g95c90f41. 2.0 MB/s | 139 kB 00:00 2024-01-15T14:31:59.769 INFO:teuthology.orchestra.run.smithi159.stdout:(17/147): libradosstriper1-18.0.0-7629.g95c90f4 5.5 MB/s | 517 kB 00:00 2024-01-15T14:31:59.837 INFO:teuthology.orchestra.run.smithi077.stdout:(5/147): ceph-mds-18.0.0-7629.g95c90f41.el8.x86 1.6 MB/s | 2.3 MB 00:01 2024-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi159.stdout:(18/147): python3-ceph-argparse-18.0.0-7629.g95 185 kB/s | 45 kB 00:00 2024-01-15T14:32:00.095 INFO:teuthology.orchestra.run.smithi159.stdout:(19/147): python3-ceph-common-18.0.0-7629.g95c9 1.5 MB/s | 128 kB 00:00 2024-01-15T14:32:00.225 INFO:teuthology.orchestra.run.smithi159.stdout:(20/147): librgw2-18.0.0-7629.g95c90f41.el8.x86 13 MB/s | 6.9 MB 00:00 2024-01-15T14:32:00.245 INFO:teuthology.orchestra.run.smithi159.stdout:(21/147): python3-cephfs-18.0.0-7629.g95c90f41. 1.3 MB/s | 197 kB 00:00 2024-01-15T14:32:00.279 INFO:teuthology.orchestra.run.smithi159.stdout:(22/147): python3-rados-18.0.0-7629.g95c90f41.e 6.8 MB/s | 370 kB 00:00 2024-01-15T14:32:00.304 INFO:teuthology.orchestra.run.smithi159.stdout:(23/147): python3-rbd-18.0.0-7629.g95c90f41.el8 6.1 MB/s | 364 kB 00:00 2024-01-15T14:32:00.312 INFO:teuthology.orchestra.run.smithi077.stdout:(6/147): ceph-mgr-18.0.0-7629.g95c90f41.el8.x86 2.1 MB/s | 1.6 MB 00:00 2024-01-15T14:32:00.329 INFO:teuthology.orchestra.run.smithi159.stdout:(24/147): python3-rgw-18.0.0-7629.g95c90f41.el8 2.1 MB/s | 107 kB 00:00 2024-01-15T14:32:00.355 INFO:teuthology.orchestra.run.smithi159.stdout:(25/147): rbd-fuse-18.0.0-7629.g95c90f41.el8.x8 1.8 MB/s | 94 kB 00:00 2024-01-15T14:32:00.538 INFO:teuthology.orchestra.run.smithi159.stdout:(26/147): rbd-nbd-18.0.0-7629.g95c90f41.el8.x86 990 kB/s | 181 kB 00:00 2024-01-15T14:32:00.639 INFO:teuthology.orchestra.run.smithi159.stdout:(27/147): rbd-mirror-18.0.0-7629.g95c90f41.el8. 12 MB/s | 3.6 MB 00:00 2024-01-15T14:32:00.664 INFO:teuthology.orchestra.run.smithi159.stdout:(28/147): ceph-grafana-dashboards-18.0.0-7629.g 223 kB/s | 28 kB 00:00 2024-01-15T14:32:00.697 INFO:teuthology.orchestra.run.smithi159.stdout:(29/147): ceph-mgr-cephadm-18.0.0-7629.g95c90f4 2.2 MB/s | 133 kB 00:00 2024-01-15T14:32:00.873 INFO:teuthology.orchestra.run.smithi159.stdout:(30/147): ceph-mgr-dashboard-18.0.0-7629.g95c90 8.4 MB/s | 1.8 MB 00:00 2024-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi159.stdout:(31/147): ceph-mgr-modules-core-18.0.0-7629.g95 2.1 MB/s | 246 kB 00:00 2024-01-15T14:32:01.240 INFO:teuthology.orchestra.run.smithi159.stdout:(32/147): ceph-mgr-rook-18.0.0-7629.g95c90f41.e 200 kB/s | 50 kB 00:00 2024-01-15T14:32:01.332 INFO:teuthology.orchestra.run.smithi159.stdout:(33/147): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2024-01-15T14:32:01.371 INFO:teuthology.orchestra.run.smithi077.stdout:(7/147): ceph-common-18.0.0-7629.g95c90f41.el8. 4.9 MB/s | 23 MB 00:04 2024-01-15T14:32:01.378 INFO:teuthology.orchestra.run.smithi159.stdout:(34/147): ceph-prometheus-alerts-18.0.0-7629.g9 109 kB/s | 15 kB 00:00 2024-01-15T14:32:01.399 INFO:teuthology.orchestra.run.smithi159.stdout:(35/147): ceph-volume-18.0.0-7629.g95c90f41.el8 3.8 MB/s | 262 kB 00:00 2024-01-15T14:32:01.458 INFO:teuthology.orchestra.run.smithi159.stdout:(36/147): cephadm-18.0.0-7629.g95c90f41.el8.noa 5.0 MB/s | 412 kB 00:00 2024-01-15T14:32:01.475 INFO:teuthology.orchestra.run.smithi159.stdout:(37/147): cephfs-top-18.0.0-7629.g95c90f41.el8. 241 kB/s | 18 kB 00:00 2024-01-15T14:32:01.859 INFO:teuthology.orchestra.run.smithi159.stdout:(38/147): liboath-2.6.2-3.el8.x86_64.rpm 153 kB/s | 59 kB 00:00 2024-01-15T14:32:01.984 INFO:teuthology.orchestra.run.smithi159.stdout:(39/147): gperftools-libs-2.7-9.el8.x86_64.rpm 583 kB/s | 306 kB 00:00 2024-01-15T14:32:02.043 INFO:teuthology.orchestra.run.smithi159.stdout:(40/147): libunwind-1.3.1-3.el8.x86_64.rpm 409 kB/s | 75 kB 00:00 2024-01-15T14:32:02.143 INFO:teuthology.orchestra.run.smithi159.stdout:(41/147): luarocks-3.5.0-1.el8.noarch.rpm 1.0 MB/s | 167 kB 00:00 2024-01-15T14:32:02.268 INFO:teuthology.orchestra.run.smithi159.stdout:(42/147): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2024-01-15T14:32:02.329 INFO:teuthology.orchestra.run.smithi077.stdout:(8/147): ceph-mon-18.0.0-7629.g95c90f41.el8.x86 1.9 MB/s | 4.6 MB 00:02 2024-01-15T14:32:02.343 INFO:teuthology.orchestra.run.smithi159.stdout:(43/147): python3-asyncssh-2.7.0-2.el8.noarch.r 1.5 MB/s | 476 kB 00:00 2024-01-15T14:32:02.377 INFO:teuthology.orchestra.run.smithi159.stdout:(44/147): python3-beautifulsoup4-4.6.3-2.el8.1. 1.7 MB/s | 185 kB 00:00 2024-01-15T14:32:02.421 INFO:teuthology.orchestra.run.smithi077.stdout:(9/147): ceph-selinux-18.0.0-7629.g95c90f41.el8 266 kB/s | 24 kB 00:00 2024-01-15T14:32:02.469 INFO:teuthology.orchestra.run.smithi159.stdout:(45/147): python3-certifi-2018.10.15-7.el8.noar 177 kB/s | 16 kB 00:00 2024-01-15T14:32:02.494 INFO:teuthology.orchestra.run.smithi159.stdout:(46/147): python3-cachetools-3.1.1-4.el8.noarch 219 kB/s | 33 kB 00:00 2024-01-15T14:32:02.636 INFO:teuthology.orchestra.run.smithi159.stdout:(47/147): python3-cheroot-8.5.2-1.el8.noarch.rp 1.0 MB/s | 173 kB 00:00 2024-01-15T14:32:02.661 INFO:teuthology.orchestra.run.smithi159.stdout:(48/147): python3-cherrypy-18.4.0-1.el8.noarch. 2.2 MB/s | 384 kB 00:00 2024-01-15T14:32:02.762 INFO:teuthology.orchestra.run.smithi159.stdout:(49/147): python3-google-auth-1.1.1-10.el8.noar 649 kB/s | 81 kB 00:00 2024-01-15T14:32:02.787 INFO:teuthology.orchestra.run.smithi159.stdout:(50/147): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-01-15T14:32:02.788 INFO:teuthology.orchestra.run.smithi077.stdout:(10/147): ceph-radosgw-18.0.0-7629.g95c90f41.el 9.8 MB/s | 14 MB 00:01 2024-01-15T14:32:02.830 INFO:teuthology.orchestra.run.smithi077.stdout:(11/147): cephfs-mirror-18.0.0-7629.g95c90f41.e 6.2 MB/s | 263 kB 00:00 2024-01-15T14:32:02.871 INFO:teuthology.orchestra.run.smithi077.stdout:(12/147): libcephfs-devel-18.0.0-7629.g95c90f41 744 kB/s | 31 kB 00:00 2024-01-15T14:32:02.912 INFO:teuthology.orchestra.run.smithi159.stdout:(51/147): python3-jaraco-functools-2.0-4.el8.no 117 kB/s | 18 kB 00:00 2024-01-15T14:32:02.980 INFO:teuthology.orchestra.run.smithi077.stdout:(13/147): libcephfs2-18.0.0-7629.g95c90f41.el8. 7.6 MB/s | 839 kB 00:00 2024-01-15T14:32:03.030 INFO:teuthology.orchestra.run.smithi077.stdout:(14/147): libcephsqlite-18.0.0-7629.g95c90f41.e 3.6 MB/s | 183 kB 00:00 2024-01-15T14:32:03.054 INFO:teuthology.orchestra.run.smithi159.stdout:(52/147): python3-logutils-0.3.5-11.el8.noarch. 344 kB/s | 49 kB 00:00 2024-01-15T14:32:03.081 INFO:teuthology.orchestra.run.smithi077.stdout:(15/147): librados-devel-18.0.0-7629.g95c90f41. 2.7 MB/s | 139 kB 00:00 2024-01-15T14:32:03.138 INFO:teuthology.orchestra.run.smithi159.stdout:(53/147): python3-kubernetes-11.0.0-6.el8.noarc 3.3 MB/s | 1.2 MB 00:00 2024-01-15T14:32:03.148 INFO:teuthology.orchestra.run.smithi077.stdout:(16/147): libradosstriper1-18.0.0-7629.g95c90f4 7.6 MB/s | 517 kB 00:00 2024-01-15T14:32:03.172 INFO:teuthology.orchestra.run.smithi159.stdout:(54/147): python3-more-itertools-7.2.0-3.el8.no 502 kB/s | 59 kB 00:00 2024-01-15T14:32:03.272 INFO:teuthology.orchestra.run.smithi159.stdout:(55/147): python3-natsort-7.1.1-2.el8.noarch.rp 451 kB/s | 60 kB 00:00 2024-01-15T14:32:03.330 INFO:teuthology.orchestra.run.smithi159.stdout:(56/147): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-01-15T14:32:03.422 INFO:teuthology.orchestra.run.smithi159.stdout:(57/147): python3-portend-2.6-1.el8.noarch.rpm 104 kB/s | 16 kB 00:00 2024-01-15T14:32:03.456 INFO:teuthology.orchestra.run.smithi159.stdout:(58/147): python3-repoze-lru-0.7-6.el8.noarch.r 268 kB/s | 34 kB 00:00 2024-01-15T14:32:03.573 INFO:teuthology.orchestra.run.smithi159.stdout:(59/147): python3-rsa-4.9-2.el8.noarch.rpm 551 kB/s | 64 kB 00:00 2024-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi159.stdout:(60/147): python3-routes-2.4.1-12.el8.noarch.rp 1.1 MB/s | 196 kB 00:00 2024-01-15T14:32:03.715 INFO:teuthology.orchestra.run.smithi159.stdout:(61/147): python3-simplegeneric-0.8.1-17.el8.no 132 kB/s | 19 kB 00:00 2024-01-15T14:32:03.741 INFO:teuthology.orchestra.run.smithi159.stdout:(62/147): python3-singledispatch-3.4.0.3-18.el8 168 kB/s | 24 kB 00:00 2024-01-15T14:32:03.748 INFO:teuthology.orchestra.run.smithi077.stdout:(17/147): librgw2-18.0.0-7629.g95c90f41.el8.x86 12 MB/s | 6.9 MB 00:00 2024-01-15T14:32:03.807 INFO:teuthology.orchestra.run.smithi077.stdout:(18/147): python3-ceph-argparse-18.0.0-7629.g95 768 kB/s | 45 kB 00:00 2024-01-15T14:32:03.841 INFO:teuthology.orchestra.run.smithi159.stdout:(63/147): python3-tempora-1.14.1-5.el8.noarch.r 233 kB/s | 29 kB 00:00 2024-01-15T14:32:03.857 INFO:teuthology.orchestra.run.smithi077.stdout:(19/147): python3-ceph-common-18.0.0-7629.g95c9 2.5 MB/s | 128 kB 00:00 2024-01-15T14:32:03.866 INFO:teuthology.orchestra.run.smithi159.stdout:(64/147): python3-trustme-0.6.0-4.el8.noarch.rp 220 kB/s | 27 kB 00:00 2024-01-15T14:32:03.899 INFO:teuthology.orchestra.run.smithi077.stdout:(20/147): python3-cephfs-18.0.0-7629.g95c90f41. 4.6 MB/s | 197 kB 00:00 2024-01-15T14:32:03.966 INFO:teuthology.orchestra.run.smithi077.stdout:(21/147): python3-rados-18.0.0-7629.g95c90f41.e 5.4 MB/s | 370 kB 00:00 2024-01-15T14:32:04.016 INFO:teuthology.orchestra.run.smithi159.stdout:(65/147): python3-waitress-1.4.3-1.el8.noarch.r 1.4 MB/s | 255 kB 00:00 2024-01-15T14:32:04.033 INFO:teuthology.orchestra.run.smithi077.stdout:(22/147): python3-rbd-18.0.0-7629.g95c90f41.el8 5.3 MB/s | 364 kB 00:00 2024-01-15T14:32:04.042 INFO:teuthology.orchestra.run.smithi159.stdout:(66/147): python3-webob-1.8.5-1.el8.1.noarch.rp 1.4 MB/s | 251 kB 00:00 2024-01-15T14:32:04.083 INFO:teuthology.orchestra.run.smithi077.stdout:(23/147): python3-rgw-18.0.0-7629.g95c90f41.el8 2.1 MB/s | 107 kB 00:00 2024-01-15T14:32:04.125 INFO:teuthology.orchestra.run.smithi077.stdout:(24/147): rbd-fuse-18.0.0-7629.g95c90f41.el8.x8 2.2 MB/s | 94 kB 00:00 2024-01-15T14:32:04.134 INFO:teuthology.orchestra.run.smithi159.stdout:(67/147): python3-websocket-client-0.56.0-5.el8 524 kB/s | 61 kB 00:00 2024-01-15T14:32:04.159 INFO:teuthology.orchestra.run.smithi159.stdout:(68/147): python3-webtest-2.0.33-1.el8.noarch.r 730 kB/s | 85 kB 00:00 2024-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi159.stdout:(69/147): python3-zc-lockfile-2.0-2.el8.noarch. 177 kB/s | 23 kB 00:00 2024-01-15T14:32:04.343 INFO:teuthology.orchestra.run.smithi159.stdout:(70/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 881 kB/s | 70 kB 00:00 2024-01-15T14:32:04.410 INFO:teuthology.orchestra.run.smithi159.stdout:(71/147): lua-devel-5.3.4-12.el8.x86_64.rpm 449 kB/s | 30 kB 00:00 2024-01-15T14:32:04.450 INFO:teuthology.orchestra.run.smithi077.stdout:(25/147): rbd-mirror-18.0.0-7629.g95c90f41.el8. 11 MB/s | 3.6 MB 00:00 2024-01-15T14:32:04.460 INFO:teuthology.orchestra.run.smithi159.stdout:(72/147): python3-jsonpointer-1.10-11.el8.noarc 400 kB/s | 20 kB 00:00 2024-01-15T14:32:04.492 INFO:teuthology.orchestra.run.smithi077.stdout:(26/147): rbd-nbd-18.0.0-7629.g95c90f41.el8.x86 4.3 MB/s | 181 kB 00:00 2024-01-15T14:32:04.510 INFO:teuthology.orchestra.run.smithi159.stdout:(73/147): python3-prettytable-0.7.2-14.el8.noar 883 kB/s | 44 kB 00:00 2024-01-15T14:32:04.526 INFO:teuthology.orchestra.run.smithi077.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7629.g 838 kB/s | 28 kB 00:00 2024-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi077.stdout:(28/147): ceph-mgr-cephadm-18.0.0-7629.g95c90f4 2.6 MB/s | 133 kB 00:00 2024-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi159.stdout:(74/147): python3-pyasn1-modules-0.3.7-6.el8.no 1.6 MB/s | 110 kB 00:00 2024-01-15T14:32:04.620 INFO:teuthology.orchestra.run.smithi159.stdout:(75/147): thrift-0.13.0-2.el8.x86_64.rpm 3.8 MB/s | 1.7 MB 00:00 2024-01-15T14:32:04.645 INFO:teuthology.orchestra.run.smithi159.stdout:(76/147): python3-jsonpatch-1.21-2.el8.noarch.r 396 kB/s | 27 kB 00:00 2024-01-15T14:32:04.703 INFO:teuthology.orchestra.run.smithi159.stdout:(77/147): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2024-01-15T14:32:04.729 INFO:teuthology.orchestra.run.smithi159.stdout:(78/147): python3-werkzeug-0.12.2-4.el8.noarch. 5.8 MB/s | 457 kB 00:00 2024-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi077.stdout:(29/147): ceph-mgr-dashboard-18.0.0-7629.g95c90 11 MB/s | 1.8 MB 00:00 2024-01-15T14:32:04.762 INFO:teuthology.orchestra.run.smithi159.stdout:(79/147): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2024-01-15T14:32:04.779 INFO:teuthology.orchestra.run.smithi159.stdout:(80/147): flex-2.6.1-9.el8.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2024-01-15T14:32:04.854 INFO:teuthology.orchestra.run.smithi159.stdout:(81/147): libtool-2.4.6-25.el8.x86_64.rpm 7.6 MB/s | 709 kB 00:00 2024-01-15T14:32:04.880 INFO:teuthology.orchestra.run.smithi159.stdout:(82/147): python3-markupsafe-0.23-19.el8.x86_64 387 kB/s | 39 kB 00:00 2024-01-15T14:32:04.905 INFO:teuthology.orchestra.run.smithi159.stdout:(83/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2024-01-15T14:32:04.931 INFO:teuthology.orchestra.run.smithi159.stdout:(84/147): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-01-15T14:32:04.956 INFO:teuthology.orchestra.run.smithi159.stdout:(85/147): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-01-15T14:32:04.973 INFO:teuthology.orchestra.run.smithi159.stdout:(86/147): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2024-01-15T14:32:05.031 INFO:teuthology.orchestra.run.smithi159.stdout:(87/147): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2024-01-15T14:32:05.149 INFO:teuthology.orchestra.run.smithi159.stdout:(88/147): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2024-01-15T14:32:05.207 INFO:teuthology.orchestra.run.smithi159.stdout:(89/147): socat-1.7.4.1-1.el8.x86_64.rpm 5.5 MB/s | 323 kB 00:00 2024-01-15T14:32:05.274 INFO:teuthology.orchestra.run.smithi159.stdout:(90/147): python3-numpy-1.14.3-10.el8.x86_64.rp 15 MB/s | 3.7 MB 00:00 2024-01-15T14:32:05.299 INFO:teuthology.orchestra.run.smithi159.stdout:(91/147): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.4 MB/s | 225 kB 00:00 2024-01-15T14:32:05.325 INFO:teuthology.orchestra.run.smithi159.stdout:(92/147): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2024-01-15T14:32:05.368 INFO:teuthology.orchestra.run.smithi077.stdout:(30/147): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2024-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi159.stdout:(93/147): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2024-01-15T14:32:05.427 INFO:teuthology.orchestra.run.smithi077.stdout:(31/147): ceph-mgr-modules-core-18.0.0-7629.g95 4.1 MB/s | 246 kB 00:00 2024-01-15T14:32:05.469 INFO:teuthology.orchestra.run.smithi077.stdout:(32/147): ceph-mgr-rook-18.0.0-7629.g95c90f41.e 1.2 MB/s | 50 kB 00:00 2024-01-15T14:32:05.492 INFO:teuthology.orchestra.run.smithi159.stdout:(94/147): openblas-threads-0.3.15-6.el8.x86_64. 27 MB/s | 5.2 MB 00:00 2024-01-15T14:32:05.511 INFO:teuthology.orchestra.run.smithi077.stdout:(33/147): ceph-prometheus-alerts-18.0.0-7629.g9 360 kB/s | 15 kB 00:00 2024-01-15T14:32:05.551 INFO:teuthology.orchestra.run.smithi159.stdout:(95/147): automake-1.16.1-8.el8.noarch.rpm 12 MB/s | 714 kB 00:00 2024-01-15T14:32:05.553 INFO:teuthology.orchestra.run.smithi077.stdout:(34/147): ceph-volume-18.0.0-7629.g95c90f41.el8 6.1 MB/s | 262 kB 00:00 2024-01-15T14:32:05.619 INFO:teuthology.orchestra.run.smithi077.stdout:(35/147): cephadm-18.0.0-7629.g95c90f41.el8.noa 6.0 MB/s | 412 kB 00:00 2024-01-15T14:32:05.626 INFO:teuthology.orchestra.run.smithi159.stdout:(96/147): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-01-15T14:32:05.651 INFO:teuthology.orchestra.run.smithi159.stdout:(97/147): libpq-13.11-1.el8.x86_64.rpm 1.9 MB/s | 198 kB 00:00 2024-01-15T14:32:05.670 INFO:teuthology.orchestra.run.smithi077.stdout:(36/147): cephfs-top-18.0.0-7629.g95c90f41.el8. 362 kB/s | 18 kB 00:00 2024-01-15T14:32:05.676 INFO:teuthology.orchestra.run.smithi159.stdout:(98/147): jq-1.6-7.el8.x86_64.rpm 3.9 MB/s | 202 kB 00:00 2024-01-15T14:32:05.723 INFO:teuthology.orchestra.run.smithi159.stdout:(99/147): python3-pytz-2017.2-11.el8.noarch.rpm 764 kB/s | 54 kB 00:00 2024-01-15T14:32:05.769 INFO:teuthology.orchestra.run.smithi159.stdout:(100/147): postgresql-10.23-3.module+el8.9.0+20 16 MB/s | 1.5 MB 00:00 2024-01-15T14:32:05.802 INFO:teuthology.orchestra.run.smithi159.stdout:(101/147): postgresql-contrib-10.23-3.module+el 10 MB/s | 811 kB 00:00 2024-01-15T14:32:05.919 INFO:teuthology.orchestra.run.smithi159.stdout:(102/147): postgresql-server-10.23-3.module+el8 34 MB/s | 5.1 MB 00:00 2024-01-15T14:32:05.962 INFO:teuthology.orchestra.run.smithi077.stdout:(37/147): ceph-osd-18.0.0-7629.g95c90f41.el8.x8 3.2 MB/s | 18 MB 00:05 2024-01-15T14:32:05.970 INFO:teuthology.orchestra.run.smithi159.stdout:(103/147): libverto-devel-0.3.0-5.el8.x86_64.rp 358 kB/s | 18 kB 00:00 2024-01-15T14:32:05.987 INFO:teuthology.orchestra.run.smithi077.stdout:(38/147): gperftools-libs-2.7-9.el8.x86_64.rpm 966 kB/s | 306 kB 00:00 2024-01-15T14:32:06.021 INFO:teuthology.orchestra.run.smithi077.stdout:(39/147): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2024-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi159.stdout:(104/147): python3-jwt-1.6.1-2.el8.noarch.rpm 641 kB/s | 43 kB 00:00 2024-01-15T14:32:06.063 INFO:teuthology.orchestra.run.smithi077.stdout:(40/147): luarocks-3.5.0-1.el8.noarch.rpm 4.0 MB/s | 167 kB 00:00 2024-01-15T14:32:06.078 INFO:teuthology.orchestra.run.smithi159.stdout:(105/147): python3-oauthlib-2.1.0-1.el8.noarch. 3.6 MB/s | 155 kB 00:00 2024-01-15T14:32:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:(41/147): liboath-2.6.2-3.el8.x86_64.rpm 439 kB/s | 59 kB 00:00 2024-01-15T14:32:06.120 INFO:teuthology.orchestra.run.smithi159.stdout:(106/147): python3-pycparser-2.14-14.el8.noarch 2.6 MB/s | 109 kB 00:00 2024-01-15T14:32:06.130 INFO:teuthology.orchestra.run.smithi077.stdout:(42/147): python3-asyncssh-2.7.0-2.el8.noarch.r 6.9 MB/s | 476 kB 00:00 2024-01-15T14:32:06.155 INFO:teuthology.orchestra.run.smithi077.stdout:(43/147): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 747 kB/s | 44 kB 00:00 2024-01-15T14:32:06.180 INFO:teuthology.orchestra.run.smithi077.stdout:(44/147): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2024-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi077.stdout:(45/147): python3-cachetools-3.1.1-4.el8.noarch 790 kB/s | 33 kB 00:00 2024-01-15T14:32:06.223 INFO:teuthology.orchestra.run.smithi077.stdout:(46/147): python3-certifi-2018.10.15-7.el8.noar 387 kB/s | 16 kB 00:00 2024-01-15T14:32:06.265 INFO:teuthology.orchestra.run.smithi077.stdout:(47/147): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2024-01-15T14:32:06.290 INFO:teuthology.orchestra.run.smithi077.stdout:(48/147): python3-cherrypy-18.4.0-1.el8.noarch. 5.6 MB/s | 384 kB 00:00 2024-01-15T14:32:06.316 INFO:teuthology.orchestra.run.smithi077.stdout:(49/147): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2024-01-15T14:32:06.338 INFO:teuthology.orchestra.run.smithi159.stdout:(107/147): python3-scipy-1.0.0-21.module+el8.9. 26 MB/s | 14 MB 00:00 2024-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi159.stdout:(108/147): python3-requests-oauthlib-1.0.0-1.el 176 kB/s | 43 kB 00:00 2024-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:(50/147): python3-jaraco-6.2-6.el8.noarch.rpm 142 kB/s | 11 kB 00:00 2024-01-15T14:32:06.380 INFO:teuthology.orchestra.run.smithi159.stdout:(109/147): libaio-devel-0.3.112-1.el8.x86_64.rp 447 kB/s | 19 kB 00:00 2024-01-15T14:32:06.391 INFO:teuthology.orchestra.run.smithi077.stdout:(51/147): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2024-01-15T14:32:06.405 INFO:teuthology.orchestra.run.smithi159.stdout:(110/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-01-15T14:32:06.425 INFO:teuthology.orchestra.run.smithi077.stdout:(52/147): python3-logutils-0.3.5-11.el8.noarch. 1.4 MB/s | 49 kB 00:00 2024-01-15T14:32:06.438 INFO:teuthology.orchestra.run.smithi159.stdout:(111/147): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2024-01-15T14:32:06.458 INFO:teuthology.orchestra.run.smithi077.stdout:(53/147): python3-kubernetes-11.0.0-6.el8.noarc 13 MB/s | 1.2 MB 00:00 2024-01-15T14:32:06.464 INFO:teuthology.orchestra.run.smithi159.stdout:(112/147): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2024-01-15T14:32:06.475 INFO:teuthology.orchestra.run.smithi077.stdout:(54/147): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2024-01-15T14:32:06.489 INFO:teuthology.orchestra.run.smithi159.stdout:(113/147): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-01-15T14:32:06.492 INFO:teuthology.orchestra.run.smithi077.stdout:(55/147): python3-natsort-7.1.1-2.el8.noarch.rp 1.8 MB/s | 60 kB 00:00 2024-01-15T14:32:06.514 INFO:teuthology.orchestra.run.smithi159.stdout:(114/147): libselinux-devel-2.9-5.el8.x86_64.rp 3.9 MB/s | 200 kB 00:00 2024-01-15T14:32:06.517 INFO:teuthology.orchestra.run.smithi077.stdout:(56/147): python3-pecan-1.3.2-9.el8.noarch.rpm 6.6 MB/s | 283 kB 00:00 2024-01-15T14:32:06.531 INFO:teuthology.orchestra.run.smithi159.stdout:(115/147): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2024-01-15T14:32:06.534 INFO:teuthology.orchestra.run.smithi077.stdout:(57/147): python3-portend-2.6-1.el8.noarch.rpm 371 kB/s | 16 kB 00:00 2024-01-15T14:32:06.557 INFO:teuthology.orchestra.run.smithi159.stdout:(116/147): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-01-15T14:32:06.559 INFO:teuthology.orchestra.run.smithi077.stdout:(58/147): python3-repoze-lru-0.7-6.el8.noarch.r 805 kB/s | 34 kB 00:00 2024-01-15T14:32:06.585 INFO:teuthology.orchestra.run.smithi077.stdout:(59/147): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2024-01-15T14:32:06.608 INFO:teuthology.orchestra.run.smithi159.stdout:(117/147): zlib-devel-1.2.11-18.el8_5.x86_64.rp 750 kB/s | 58 kB 00:00 2024-01-15T14:32:06.610 INFO:teuthology.orchestra.run.smithi077.stdout:(60/147): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-01-15T14:32:06.635 INFO:teuthology.orchestra.run.smithi077.stdout:(61/147): python3-simplegeneric-0.8.1-17.el8.no 374 kB/s | 19 kB 00:00 2024-01-15T14:32:06.649 INFO:teuthology.orchestra.run.smithi159.stdout:(118/147): libcom_err-devel-1.45.6-4.el8.x86_64 419 kB/s | 39 kB 00:00 2024-01-15T14:32:06.660 INFO:teuthology.orchestra.run.smithi077.stdout:(62/147): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2024-01-15T14:32:06.674 INFO:teuthology.orchestra.run.smithi159.stdout:(119/147): fuse-2.9.7-17.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-01-15T14:32:06.686 INFO:teuthology.orchestra.run.smithi077.stdout:(63/147): python3-tempora-1.14.1-5.el8.noarch.r 583 kB/s | 29 kB 00:00 2024-01-15T14:32:06.708 INFO:teuthology.orchestra.run.smithi159.stdout:(120/147): libgfortran-8.5.0-20.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2024-01-15T14:32:06.711 INFO:teuthology.orchestra.run.smithi077.stdout:(64/147): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2024-01-15T14:32:06.733 INFO:teuthology.orchestra.run.smithi159.stdout:(121/147): libquadmath-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-01-15T14:32:06.736 INFO:teuthology.orchestra.run.smithi077.stdout:(65/147): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2024-01-15T14:32:06.758 INFO:teuthology.orchestra.run.smithi159.stdout:(122/147): libuuid-devel-2.32.1-43.el8.x86_64.r 1.9 MB/s | 100 kB 00:00 2024-01-15T14:32:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:(66/147): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2024-01-15T14:32:06.784 INFO:teuthology.orchestra.run.smithi159.stdout:(123/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2024-01-15T14:32:06.795 INFO:teuthology.orchestra.run.smithi077.stdout:(67/147): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2024-01-15T14:32:06.818 INFO:teuthology.orchestra.run.smithi159.stdout:(124/147): python3-cryptography-3.2.1-6.el8.x86 9.3 MB/s | 559 kB 00:00 2024-01-15T14:32:06.820 INFO:teuthology.orchestra.run.smithi077.stdout:(68/147): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2024-01-15T14:32:06.843 INFO:teuthology.orchestra.run.smithi159.stdout:(125/147): xfsprogs-devel-5.0.0-12.el8.x86_64.r 1.2 MB/s | 73 kB 00:00 2024-01-15T14:32:06.846 INFO:teuthology.orchestra.run.smithi077.stdout:(69/147): python3-zc-lockfile-2.0-2.el8.noarch. 459 kB/s | 23 kB 00:00 2024-01-15T14:32:06.886 INFO:teuthology.orchestra.run.smithi159.stdout:(126/147): elfutils-libelf-devel-0.189-3.el8.x8 902 kB/s | 62 kB 00:00 2024-01-15T14:32:06.888 INFO:teuthology.orchestra.run.smithi077.stdout:(70/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2024-01-15T14:32:06.918 INFO:teuthology.orchestra.run.smithi159.stdout:(127/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-01-15T14:32:06.938 INFO:teuthology.orchestra.run.smithi077.stdout:(71/147): libverto-devel-0.3.0-5.el8.x86_64.rpm 359 kB/s | 18 kB 00:00 2024-01-15T14:32:06.952 INFO:teuthology.orchestra.run.smithi159.stdout:(128/147): krb5-devel-1.18.2-26.el8_9.x86_64.rp 8.4 MB/s | 562 kB 00:00 2024-01-15T14:32:06.988 INFO:teuthology.orchestra.run.smithi077.stdout:(72/147): thrift-0.13.0-2.el8.x86_64.rpm 10 MB/s | 1.7 MB 00:00 2024-01-15T14:32:07.002 INFO:teuthology.orchestra.run.smithi159.stdout:(129/147): NetworkManager-initscripts-updown-1. 2.8 MB/s | 144 kB 00:00 2024-01-15T14:32:07.014 INFO:teuthology.orchestra.run.smithi077.stdout:(73/147): python3-jwt-1.6.1-2.el8.noarch.rpm 569 kB/s | 43 kB 00:00 2024-01-15T14:32:07.044 INFO:teuthology.orchestra.run.smithi159.stdout:(130/147): openssl-devel-1.1.1k-12.el8_9.x86_64 19 MB/s | 2.3 MB 00:00 2024-01-15T14:32:07.056 INFO:teuthology.orchestra.run.smithi077.stdout:(74/147): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2024-01-15T14:32:07.081 INFO:teuthology.orchestra.run.smithi077.stdout:(75/147): python3-pycparser-2.14-14.el8.noarch. 1.6 MB/s | 109 kB 00:00 2024-01-15T14:32:07.106 INFO:teuthology.orchestra.run.smithi077.stdout:(76/147): python3-requests-oauthlib-1.0.0-1.el8 848 kB/s | 43 kB 00:00 2024-01-15T14:32:07.132 INFO:teuthology.orchestra.run.smithi077.stdout:(77/147): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2024-01-15T14:32:07.157 INFO:teuthology.orchestra.run.smithi077.stdout:(78/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 872 kB/s | 44 kB 00:00 2024-01-15T14:32:07.190 INFO:teuthology.orchestra.run.smithi077.stdout:(79/147): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2024-01-15T14:32:07.216 INFO:teuthology.orchestra.run.smithi077.stdout:(80/147): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2024-01-15T14:32:07.241 INFO:teuthology.orchestra.run.smithi077.stdout:(81/147): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-01-15T14:32:07.266 INFO:teuthology.orchestra.run.smithi077.stdout:(82/147): libselinux-devel-2.9-5.el8.x86_64.rpm 3.9 MB/s | 200 kB 00:00 2024-01-15T14:32:07.292 INFO:teuthology.orchestra.run.smithi077.stdout:(83/147): keyutils-libs-devel-1.5.10-9.el8.x86_ 959 kB/s | 48 kB 00:00 2024-01-15T14:32:07.309 INFO:teuthology.orchestra.run.smithi077.stdout:(84/147): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-01-15T14:32:07.334 INFO:teuthology.orchestra.run.smithi077.stdout:(85/147): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2024-01-15T14:32:07.359 INFO:teuthology.orchestra.run.smithi077.stdout:(86/147): libcom_err-devel-1.45.6-4.el8.x86_64. 770 kB/s | 39 kB 00:00 2024-01-15T14:32:07.378 INFO:teuthology.orchestra.run.smithi159.stdout:(131/147): librados2-18.0.0-7629.g95c90f41.el8. 10 MB/s | 3.8 MB 00:00 2024-01-15T14:32:07.409 INFO:teuthology.orchestra.run.smithi077.stdout:(87/147): fuse-2.9.7-17.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2024-01-15T14:32:07.428 INFO:teuthology.orchestra.run.smithi159.stdout:(132/147): elfutils-libelf-0.189-3.el8.x86_64.r 4.6 MB/s | 232 kB 00:00 2024-01-15T14:32:07.443 INFO:teuthology.orchestra.run.smithi077.stdout:(88/147): libgfortran-8.5.0-20.el8.x86_64.rpm 7.5 MB/s | 645 kB 00:00 2024-01-15T14:32:07.469 INFO:teuthology.orchestra.run.smithi077.stdout:(89/147): libquadmath-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-01-15T14:32:07.494 INFO:teuthology.orchestra.run.smithi077.stdout:(90/147): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 100 kB 00:00 2024-01-15T14:32:07.503 INFO:teuthology.orchestra.run.smithi159.stdout:(133/147): librbd1-18.0.0-7629.g95c90f41.el8.x8 8.1 MB/s | 3.7 MB 00:00 2024-01-15T14:32:07.519 INFO:teuthology.orchestra.run.smithi077.stdout:(91/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2024-01-15T14:32:07.529 INFO:teuthology.orchestra.run.smithi159.stdout:(134/147): elfutils-libs-0.189-3.el8.x86_64.rpm 3.0 MB/s | 303 kB 00:00 2024-01-15T14:32:07.552 INFO:teuthology.orchestra.run.smithi077.stdout:(92/147): python3-cryptography-3.2.1-6.el8.x86_ 9.3 MB/s | 559 kB 00:00 2024-01-15T14:32:07.554 INFO:teuthology.orchestra.run.smithi159.stdout:(135/147): libmount-2.32.1-43.el8.x86_64.rpm 4.6 MB/s | 237 kB 00:00 2024-01-15T14:32:07.571 INFO:teuthology.orchestra.run.smithi159.stdout:(136/147): libsmartcols-2.32.1-43.el8.x86_64.rp 4.2 MB/s | 179 kB 00:00 2024-01-15T14:32:07.578 INFO:teuthology.orchestra.run.smithi077.stdout:(93/147): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.2 MB/s | 73 kB 00:00 2024-01-15T14:32:07.603 INFO:teuthology.orchestra.run.smithi077.stdout:(94/147): elfutils-libelf-devel-0.189-3.el8.x86 1.2 MB/s | 62 kB 00:00 2024-01-15T14:32:07.604 INFO:teuthology.orchestra.run.smithi159.stdout:(137/147): libuuid-2.32.1-43.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-01-15T14:32:07.620 INFO:teuthology.orchestra.run.smithi077.stdout:(95/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-01-15T14:32:07.653 INFO:teuthology.orchestra.run.smithi077.stdout:(96/147): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 11 MB/s | 562 kB 00:00 2024-01-15T14:32:07.671 INFO:teuthology.orchestra.run.smithi159.stdout:(138/147): util-linux-2.32.1-43.el8.x86_64.rpm 25 MB/s | 2.5 MB 00:00 2024-01-15T14:32:07.696 INFO:teuthology.orchestra.run.smithi077.stdout:(97/147): NetworkManager-initscripts-updown-1.4 3.4 MB/s | 144 kB 00:00 2024-01-15T14:32:07.697 INFO:teuthology.orchestra.run.smithi159.stdout:(139/147): util-linux-user-2.32.1-43.el8.x86_64 1.1 MB/s | 103 kB 00:00 2024-01-15T14:32:07.729 INFO:teuthology.orchestra.run.smithi077.stdout:(98/147): python3-jsonpointer-1.10-11.el8.noarc 602 kB/s | 20 kB 00:00 2024-01-15T14:32:07.764 INFO:teuthology.orchestra.run.smithi159.stdout:(140/147): iproute-6.2.0-5.el8_9.x86_64.rpm 9.1 MB/s | 854 kB 00:00 2024-01-15T14:32:07.771 INFO:teuthology.orchestra.run.smithi077.stdout:(99/147): openssl-devel-1.1.1k-12.el8_9.x86_64. 15 MB/s | 2.3 MB 00:00 2024-01-15T14:32:07.796 INFO:teuthology.orchestra.run.smithi077.stdout:(100/147): python3-prettytable-0.7.2-14.el8.noa 659 kB/s | 44 kB 00:00 2024-01-15T14:32:07.797 INFO:teuthology.orchestra.run.smithi159.stdout:(141/147): libblkid-2.32.1-43.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2024-01-15T14:32:07.822 INFO:teuthology.orchestra.run.smithi077.stdout:(101/147): python3-pyasn1-modules-0.3.7-6.el8.n 2.1 MB/s | 110 kB 00:00 2024-01-15T14:32:07.822 INFO:teuthology.orchestra.run.smithi159.stdout:(142/147): libfdisk-2.32.1-43.el8.x86_64.rpm 4.2 MB/s | 254 kB 00:00 2024-01-15T14:32:07.847 INFO:teuthology.orchestra.run.smithi077.stdout:(102/147): python3-jsonpatch-1.21-2.el8.noarch. 527 kB/s | 27 kB 00:00 2024-01-15T14:32:07.864 INFO:teuthology.orchestra.run.smithi159.stdout:(143/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-01-15T14:32:07.872 INFO:teuthology.orchestra.run.smithi077.stdout:(103/147): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2024-01-15T14:32:07.898 INFO:teuthology.orchestra.run.smithi159.stdout:(144/147): openssl-1.1.1k-12.el8_9.x86_64.rpm 9.2 MB/s | 711 kB 00:00 2024-01-15T14:32:07.931 INFO:teuthology.orchestra.run.smithi077.stdout:(104/147): python3-werkzeug-0.12.2-4.el8.noarch 5.4 MB/s | 457 kB 00:00 2024-01-15T14:32:07.948 INFO:teuthology.orchestra.run.smithi159.stdout:(145/147): openssl-libs-1.1.1k-12.el8_9.x86_64. 18 MB/s | 1.5 MB 00:00 2024-01-15T14:32:07.973 INFO:teuthology.orchestra.run.smithi077.stdout:(105/147): bison-3.0.4-10.el8.x86_64.rpm 6.7 MB/s | 688 kB 00:00 2024-01-15T14:32:07.974 INFO:teuthology.orchestra.run.smithi159.stdout:(146/147): xfsdump-3.1.8-7.el8_9.x86_64.rpm 4.3 MB/s | 333 kB 00:00 2024-01-15T14:32:08.007 INFO:teuthology.orchestra.run.smithi077.stdout:(106/147): flex-2.6.1-9.el8.x86_64.rpm 4.1 MB/s | 320 kB 00:00 2024-01-15T14:32:08.048 INFO:teuthology.orchestra.run.smithi077.stdout:(107/147): libtool-2.4.6-25.el8.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2024-01-15T14:32:08.074 INFO:teuthology.orchestra.run.smithi077.stdout:(108/147): python3-markupsafe-0.23-19.el8.x86_6 581 kB/s | 39 kB 00:00 2024-01-15T14:32:08.099 INFO:teuthology.orchestra.run.smithi077.stdout:(109/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2024-01-15T14:32:08.116 INFO:teuthology.orchestra.run.smithi077.stdout:(110/147): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-01-15T14:32:08.141 INFO:teuthology.orchestra.run.smithi077.stdout:(111/147): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi077.stdout:(112/147): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2024-01-15T14:32:08.225 INFO:teuthology.orchestra.run.smithi077.stdout:(113/147): python3-jinja2-2.10.1-3.el8.noarch.r 9.0 MB/s | 538 kB 00:00 2024-01-15T14:32:08.342 INFO:teuthology.orchestra.run.smithi077.stdout:(114/147): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2024-01-15T14:32:08.384 INFO:teuthology.orchestra.run.smithi077.stdout:(115/147): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2024-01-15T14:32:08.459 INFO:teuthology.orchestra.run.smithi077.stdout:(116/147): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2024-01-15T14:32:08.485 INFO:teuthology.orchestra.run.smithi077.stdout:(117/147): python3-numpy-f2py-1.14.3-10.el8.x86 2.2 MB/s | 225 kB 00:00 2024-01-15T14:32:08.518 INFO:teuthology.orchestra.run.smithi077.stdout:(118/147): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2024-01-15T14:32:08.560 INFO:teuthology.orchestra.run.smithi077.stdout:(119/147): python3-mako-1.0.6-14.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2024-01-15T14:32:08.677 INFO:teuthology.orchestra.run.smithi077.stdout:(120/147): openblas-threads-0.3.15-6.el8.x86_64 27 MB/s | 5.2 MB 00:00 2024-01-15T14:32:08.728 INFO:teuthology.orchestra.run.smithi077.stdout:(121/147): automake-1.16.1-8.el8.noarch.rpm 14 MB/s | 714 kB 00:00 2024-01-15T14:32:08.794 INFO:teuthology.orchestra.run.smithi077.stdout:(122/147): openblas-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-01-15T14:32:08.820 INFO:teuthology.orchestra.run.smithi077.stdout:(123/147): libpq-13.11-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2024-01-15T14:32:08.837 INFO:teuthology.orchestra.run.smithi077.stdout:(124/147): jq-1.6-7.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2024-01-15T14:32:08.854 INFO:teuthology.orchestra.run.smithi077.stdout:(125/147): python3-pytz-2017.2-11.el8.noarch.rp 1.6 MB/s | 54 kB 00:00 2024-01-15T14:32:08.924 INFO:teuthology.orchestra.run.smithi077.stdout:(126/147): postgresql-10.23-3.module+el8.9.0+20 17 MB/s | 1.5 MB 00:00 2024-01-15T14:32:08.962 INFO:teuthology.orchestra.run.smithi077.stdout:(127/147): postgresql-contrib-10.23-3.module+el 7.3 MB/s | 811 kB 00:00 2024-01-15T14:32:09.080 INFO:teuthology.orchestra.run.smithi077.stdout:(128/147): postgresql-server-10.23-3.module+el8 33 MB/s | 5.1 MB 00:00 2024-01-15T14:32:09.105 INFO:teuthology.orchestra.run.smithi077.stdout:(129/147): lua-devel-5.3.4-12.el8.x86_64.rpm 1.2 MB/s | 30 kB 00:00 2024-01-15T14:32:09.305 INFO:teuthology.orchestra.run.smithi077.stdout:(130/147): python3-scipy-1.0.0-21.module+el8.9. 40 MB/s | 14 MB 00:00 2024-01-15T14:32:09.689 INFO:teuthology.orchestra.run.smithi077.stdout:(131/147): librbd1-18.0.0-7629.g95c90f41.el8.x8 9.7 MB/s | 3.7 MB 00:00 2024-01-15T14:32:09.730 INFO:teuthology.orchestra.run.smithi077.stdout:(132/147): elfutils-libelf-0.189-3.el8.x86_64.r 5.5 MB/s | 232 kB 00:00 2024-01-15T14:32:09.773 INFO:teuthology.orchestra.run.smithi077.stdout:(133/147): elfutils-libs-0.189-3.el8.x86_64.rpm 7.1 MB/s | 303 kB 00:00 2024-01-15T14:32:09.815 INFO:teuthology.orchestra.run.smithi077.stdout:(134/147): libmount-2.32.1-43.el8.x86_64.rpm 5.5 MB/s | 237 kB 00:00 2024-01-15T14:32:09.848 INFO:teuthology.orchestra.run.smithi077.stdout:(135/147): libsmartcols-2.32.1-43.el8.x86_64.rp 5.3 MB/s | 179 kB 00:00 2024-01-15T14:32:09.882 INFO:teuthology.orchestra.run.smithi077.stdout:(136/147): libuuid-2.32.1-43.el8.x86_64.rpm 2.9 MB/s | 99 kB 00:00 2024-01-15T14:32:09.991 INFO:teuthology.orchestra.run.smithi077.stdout:(137/147): util-linux-2.32.1-43.el8.x86_64.rpm 23 MB/s | 2.5 MB 00:00 2024-01-15T14:32:10.024 INFO:teuthology.orchestra.run.smithi077.stdout:(138/147): util-linux-user-2.32.1-43.el8.x86_64 3.0 MB/s | 103 kB 00:00 2024-01-15T14:32:10.083 INFO:teuthology.orchestra.run.smithi077.stdout:(139/147): iproute-6.2.0-5.el8_9.x86_64.rpm 14 MB/s | 854 kB 00:00 2024-01-15T14:32:10.125 INFO:teuthology.orchestra.run.smithi077.stdout:(140/147): libblkid-2.32.1-43.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-01-15T14:32:10.166 INFO:teuthology.orchestra.run.smithi077.stdout:(141/147): libfdisk-2.32.1-43.el8.x86_64.rpm 5.9 MB/s | 254 kB 00:00 2024-01-15T14:32:10.225 INFO:teuthology.orchestra.run.smithi077.stdout:(142/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-01-15T14:32:10.275 INFO:teuthology.orchestra.run.smithi077.stdout:(143/147): openssl-1.1.1k-12.el8_9.x86_64.rpm 14 MB/s | 711 kB 00:00 2024-01-15T14:32:10.559 INFO:teuthology.orchestra.run.smithi077.stdout:(144/147): openssl-libs-1.1.1k-12.el8_9.x86_64. 5.2 MB/s | 1.5 MB 00:00 2024-01-15T14:32:10.618 INFO:teuthology.orchestra.run.smithi077.stdout:(145/147): librados2-18.0.0-7629.g95c90f41.el8. 2.5 MB/s | 3.8 MB 00:01 2024-01-15T14:32:10.651 INFO:teuthology.orchestra.run.smithi077.stdout:(146/147): xfsdump-3.1.8-7.el8_9.x86_64.rpm 3.6 MB/s | 333 kB 00:00 2024-01-15T14:32:16.115 INFO:teuthology.orchestra.run.smithi159.stdout:(147/147): ceph-test-18.0.0-7629.g95c90f41.el8. 2.8 MB/s | 49 MB 00:17 2024-01-15T14:32:16.118 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-01-15T14:32:16.118 INFO:teuthology.orchestra.run.smithi159.stdout:Total 11 MB/s | 218 MB 00:20 2024-01-15T14:32:17.154 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-01-15T14:32:17.555 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-01-15T14:32:17.555 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-01-15T14:32:17.968 INFO:teuthology.orchestra.run.smithi077.stdout:(147/147): ceph-test-18.0.0-7629.g95c90f41.el8. 3.2 MB/s | 49 MB 00:15 2024-01-15T14:32:17.973 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-01-15T14:32:17.973 INFO:teuthology.orchestra.run.smithi077.stdout:Total 10 MB/s | 218 MB 00:21 2024-01-15T14:32:18.771 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-01-15T14:32:18.782 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-01-15T14:32:19.014 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-01-15T14:32:19.409 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-01-15T14:32:19.409 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-01-15T14:32:20.626 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-01-15T14:32:20.637 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-01-15T14:32:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-01-15T14:32:24.949 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-15T14:32:25.226 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T14:32:25.438 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T14:32:25.620 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/163 2024-01-15T14:32:25.806 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T14:32:25.882 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T14:32:26.064 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T14:32:26.135 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T14:32:26.379 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/163 2024-01-15T14:32:26.523 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/163 2024-01-15T14:32:26.585 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/163 2024-01-15T14:32:26.696 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/163 2024-01-15T14:32:26.831 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-15T14:32:26.973 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T14:32:27.078 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T14:32:27.290 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-15T14:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T14:32:27.487 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/163 2024-01-15T14:32:27.623 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T14:32:27.634 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T14:32:27.711 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-15T14:32:27.868 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T14:32:27.938 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-15T14:32:28.039 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T14:32:28.139 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/163 2024-01-15T14:32:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T14:32:28.316 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/163 2024-01-15T14:32:28.356 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T14:32:28.501 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/163 2024-01-15T14:32:28.514 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T14:32:28.605 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/163 2024-01-15T14:32:28.613 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T14:32:28.784 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T14:32:29.192 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-15T14:32:29.376 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T14:32:29.389 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T14:32:29.497 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T14:32:29.698 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T14:32:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T14:32:29.930 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T14:32:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T14:32:30.015 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T14:32:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T14:32:30.211 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpq-13.11-1.el8.x86_64 15/163 2024-01-15T14:32:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T14:32:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T14:32:30.400 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 16/163 2024-01-15T14:32:30.600 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/163 2024-01-15T14:32:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 18/163 2024-01-15T14:32:31.117 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 19/163 2024-01-15T14:32:31.234 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T14:32:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T14:32:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T14:32:31.638 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T14:32:31.658 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T14:32:31.703 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T14:32:31.744 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-15T14:32:31.913 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 22/163 2024-01-15T14:32:32.018 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpq-13.11-1.el8.x86_64 15/163 2024-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 22/163 2024-01-15T14:32:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 23/163 2024-01-15T14:32:32.315 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 16/163 2024-01-15T14:32:32.488 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/163 2024-01-15T14:32:32.829 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 23/163 2024-01-15T14:32:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 18/163 2024-01-15T14:32:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 24/163 2024-01-15T14:32:33.133 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 19/163 2024-01-15T14:32:33.267 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T14:32:33.390 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 20/163 2024-01-15T14:32:33.525 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T14:32:33.627 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 21/163 2024-01-15T14:32:33.661 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 24/163 2024-01-15T14:32:33.796 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 22/163 2024-01-15T14:32:33.855 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.15-6.el8.x86_64 25/163 2024-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/163 2024-01-15T14:32:34.348 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/163 2024-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 25/163 2024-01-15T14:32:34.547 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 25/163 2024-01-15T14:32:34.673 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 25/163 2024-01-15T14:32:34.816 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 26/163 2024-01-15T14:32:34.827 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 26/163 2024-01-15T14:32:34.960 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 27/163 2024-01-15T14:32:35.158 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 28/163 2024-01-15T14:32:35.468 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 26/163 2024-01-15T14:32:35.481 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 29/163 2024-01-15T14:32:35.661 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 27/163 2024-01-15T14:32:35.776 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 30/163 2024-01-15T14:32:36.047 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 31/163 2024-01-15T14:32:36.211 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 32/163 2024-01-15T14:32:36.300 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 27/163 2024-01-15T14:32:36.378 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T14:32:36.475 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-6.el8.x86_64 28/163 2024-01-15T14:32:36.639 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 34/163 2024-01-15T14:32:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 28/163 2024-01-15T14:32:37.481 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 29/163 2024-01-15T14:32:37.617 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 35/163 2024-01-15T14:32:37.682 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 30/163 2024-01-15T14:32:37.960 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 36/163 2024-01-15T14:32:37.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 31/163 2024-01-15T14:32:38.132 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 32/163 2024-01-15T14:32:38.258 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 37/163 2024-01-15T14:32:38.331 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T14:32:38.437 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 38/163 2024-01-15T14:32:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 34/163 2024-01-15T14:32:38.799 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/163 2024-01-15T14:32:40.173 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 38/163 2024-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 39/163 2024-01-15T14:32:40.611 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/163 2024-01-15T14:32:40.774 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/163 2024-01-15T14:32:40.965 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 41/163 2024-01-15T14:32:40.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 36/163 2024-01-15T14:32:41.189 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 42/163 2024-01-15T14:32:41.376 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 43/163 2024-01-15T14:32:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 44/163 2024-01-15T14:32:41.812 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T14:32:41.873 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T14:32:41.967 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/163 2024-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T14:32:42.199 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T14:32:42.309 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/163 2024-01-15T14:32:42.433 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/163 2024-01-15T14:32:42.550 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 39/163 2024-01-15T14:32:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 48/163 2024-01-15T14:32:42.770 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/163 2024-01-15T14:32:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T14:32:42.970 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 41/163 2024-01-15T14:32:43.186 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 42/163 2024-01-15T14:32:43.380 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 43/163 2024-01-15T14:32:43.443 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T14:32:43.596 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 44/163 2024-01-15T14:32:43.779 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T14:32:43.782 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T14:32:43.795 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T14:32:43.858 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/163 2024-01-15T14:32:44.016 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.7-17.el8.x86_64 51/163 2024-01-15T14:32:44.091 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T14:32:44.158 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-15T14:32:44.242 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 52/163 2024-01-15T14:32:44.378 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/163 2024-01-15T14:32:44.399 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/163 2024-01-15T14:32:44.606 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/163 2024-01-15T14:32:44.621 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 48/163 2024-01-15T14:32:44.848 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/163 2024-01-15T14:32:44.852 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T14:32:45.189 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 56/163 2024-01-15T14:32:45.290 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-15T14:32:45.458 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 57/163 2024-01-15T14:32:45.607 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T14:32:45.618 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-15T14:32:45.691 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/163 2024-01-15T14:32:45.823 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 59/163 2024-01-15T14:32:45.847 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 51/163 2024-01-15T14:32:46.055 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 60/163 2024-01-15T14:32:46.108 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.1-8.el8.noarch 52/163 2024-01-15T14:32:46.290 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-43.el8.x86_64 53/163 2024-01-15T14:32:46.376 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 53/163 2024-01-15T14:32:46.435 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 61/163 2024-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 54/163 2024-01-15T14:32:46.658 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 54/163 2024-01-15T14:32:46.839 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 62/163 2024-01-15T14:32:46.840 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-7.el8.x86_64 55/163 2024-01-15T14:32:46.988 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 56/163 2024-01-15T14:32:47.029 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 63/163 2024-01-15T14:32:47.197 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 57/163 2024-01-15T14:32:47.291 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 64/163 2024-01-15T14:32:47.375 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 58/163 2024-01-15T14:32:47.574 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 65/163 2024-01-15T14:32:47.594 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-17.el8.x86_64 59/163 2024-01-15T14:32:47.802 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 60/163 2024-01-15T14:32:47.928 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 66/163 2024-01-15T14:32:47.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/163 2024-01-15T14:32:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/163 2024-01-15T14:32:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 67/163 2024-01-15T14:32:48.328 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/163 2024-01-15T14:32:48.370 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 68/163 2024-01-15T14:32:48.597 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 69/163 2024-01-15T14:32:48.615 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 64/163 2024-01-15T14:32:48.776 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 70/163 2024-01-15T14:32:48.912 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 65/163 2024-01-15T14:32:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 71/163 2024-01-15T14:32:49.145 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 66/163 2024-01-15T14:32:49.311 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 67/163 2024-01-15T14:32:49.337 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : automake-1.16.1-8.el8.noarch 72/163 2024-01-15T14:32:49.518 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 68/163 2024-01-15T14:32:49.552 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : uuid-1.6.2-43.el8.x86_64 73/163 2024-01-15T14:32:49.637 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 73/163 2024-01-15T14:32:49.861 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 74/163 2024-01-15T14:32:49.937 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 69/163 2024-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 74/163 2024-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-7.el8.x86_64 75/163 2024-01-15T14:32:50.225 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 70/163 2024-01-15T14:32:50.318 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 76/163 2024-01-15T14:32:50.394 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 71/163 2024-01-15T14:32:50.460 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 77/163 2024-01-15T14:32:50.612 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 72/163 2024-01-15T14:32:50.640 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 78/163 2024-01-15T14:32:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 79/163 2024-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/163 2024-01-15T14:32:50.961 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 80/163 2024-01-15T14:32:51.149 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 81/163 2024-01-15T14:32:51.192 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/163 2024-01-15T14:32:51.406 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 75/163 2024-01-15T14:32:51.416 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 82/163 2024-01-15T14:32:51.574 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 76/163 2024-01-15T14:32:51.659 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 83/163 2024-01-15T14:32:51.792 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 77/163 2024-01-15T14:32:51.873 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 84/163 2024-01-15T14:32:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 78/163 2024-01-15T14:32:52.080 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 85/163 2024-01-15T14:32:52.241 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 79/163 2024-01-15T14:32:52.287 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 86/163 2024-01-15T14:32:52.448 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 80/163 2024-01-15T14:32:52.541 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 87/163 2024-01-15T14:32:52.626 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 81/163 2024-01-15T14:32:52.770 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 88/163 2024-01-15T14:32:52.900 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 82/163 2024-01-15T14:32:52.964 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 89/163 2024-01-15T14:32:53.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 83/163 2024-01-15T14:32:53.124 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 90/163 2024-01-15T14:32:53.319 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 84/163 2024-01-15T14:32:53.525 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 85/163 2024-01-15T14:32:53.754 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 86/163 2024-01-15T14:32:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 91/163 2024-01-15T14:32:53.983 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 87/163 2024-01-15T14:32:54.024 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 92/163 2024-01-15T14:32:54.193 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 88/163 2024-01-15T14:32:54.216 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 93/163 2024-01-15T14:32:54.393 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 89/163 2024-01-15T14:32:54.475 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-2.6-1.el8.noarch 94/163 2024-01-15T14:32:54.570 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 90/163 2024-01-15T14:32:54.742 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 95/163 2024-01-15T14:32:54.784 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 91/163 2024-01-15T14:32:54.917 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 96/163 2024-01-15T14:32:55.007 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 92/163 2024-01-15T14:32:55.168 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 97/163 2024-01-15T14:32:55.170 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 93/163 2024-01-15T14:32:55.475 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 94/163 2024-01-15T14:32:55.722 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 95/163 2024-01-15T14:32:55.945 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 96/163 2024-01-15T14:32:56.145 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 97/163 2024-01-15T14:32:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 98/163 2024-01-15T14:32:56.580 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 99/163 2024-01-15T14:32:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 100/163 2024-01-15T14:32:57.126 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 101/163 2024-01-15T14:32:57.241 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 98/163 2024-01-15T14:32:57.445 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 102/163 2024-01-15T14:32:57.752 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.5.0-1.el8.noarch 103/163 2024-01-15T14:32:57.984 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 104/163 2024-01-15T14:32:58.060 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 99/163 2024-01-15T14:32:58.152 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 105/163 2024-01-15T14:32:58.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 100/163 2024-01-15T14:32:58.610 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 101/163 2024-01-15T14:32:58.933 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 102/163 2024-01-15T14:32:59.197 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.5.0-1.el8.noarch 103/163 2024-01-15T14:32:59.388 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 104/163 2024-01-15T14:32:59.606 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 105/163 2024-01-15T14:33:02.565 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T14:33:02.826 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T14:33:03.335 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T14:33:03.400 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T14:33:03.695 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T14:33:03.806 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T14:33:03.959 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T14:33:03.964 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 106/163 2024-01-15T14:33:04.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T14:33:04.908 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 107/163 2024-01-15T14:33:05.046 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /run 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-01-15T14:33:23.289 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-01-15T14:33:23.290 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2024-01-15T14:33:23.290 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:33:23.623 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 108/163 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-01-15T14:33:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-01-15T14:33:24.518 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-01-15T14:33:24.518 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-01-15T14:33:24.518 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2024-01-15T14:33:24.518 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:33:24.869 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T14:33:25.221 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T14:33:25.663 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T14:33:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T14:33:26.281 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T14:33:26.431 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T14:33:26.449 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T14:33:26.462 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 109/163 2024-01-15T14:33:26.646 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T14:33:26.866 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 110/163 2024-01-15T14:33:27.035 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T14:33:27.493 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 111/163 2024-01-15T14:33:27.636 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T14:33:27.653 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 112/163 2024-01-15T14:33:28.457 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T14:33:28.644 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T14:33:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 114/163 2024-01-15T14:33:28.921 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 115/163 2024-01-15T14:33:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T14:33:29.346 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T14:33:29.392 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T14:33:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 113/163 2024-01-15T14:33:29.682 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T14:33:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 114/163 2024-01-15T14:33:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 115/163 2024-01-15T14:33:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T14:33:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 116/163 2024-01-15T14:33:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T14:33:30.647 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T14:33:30.829 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T14:33:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T14:33:31.275 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 119/163 2024-01-15T14:33:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T14:33:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T14:33:31.597 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 117/163 2024-01-15T14:33:31.646 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T14:33:31.658 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T14:33:31.759 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T14:33:31.852 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 118/163 2024-01-15T14:33:32.195 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 119/163 2024-01-15T14:33:32.337 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T14:33:32.374 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 120/163 2024-01-15T14:33:32.518 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T14:33:32.529 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 121/163 2024-01-15T14:33:33.020 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 122/163 2024-01-15T14:33:33.182 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T14:33:33.217 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T14:33:33.403 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T14:33:33.446 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T14:33:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T14:33:33.878 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 122/163 2024-01-15T14:33:33.974 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T14:33:34.037 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T14:33:34.071 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 123/163 2024-01-15T14:33:34.171 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T14:33:34.271 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T14:33:34.313 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g95c90 124/163 2024-01-15T14:33:34.451 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T14:33:34.811 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 125/163 2024-01-15T14:33:34.968 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T14:33:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T14:33:39.287 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 127/163 2024-01-15T14:33:39.619 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 126/163 2024-01-15T14:33:39.621 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 128/163 2024-01-15T14:33:39.845 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libtool-2.4.6-25.el8.x86_64 129/163 2024-01-15T14:33:40.523 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 129/163 2024-01-15T14:33:40.577 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 127/163 2024-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/163 2024-01-15T14:33:40.913 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/163 2024-01-15T14:33:41.115 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 131/163 2024-01-15T14:33:41.213 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/163 2024-01-15T14:33:41.292 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 132/163 2024-01-15T14:33:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/163 2024-01-15T14:33:41.600 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 130/163 2024-01-15T14:33:41.732 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/163 2024-01-15T14:33:41.888 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 135/163 2024-01-15T14:33:41.894 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 131/163 2024-01-15T14:33:42.066 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-25.el8.x86_64 132/163 2024-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 132/163 2024-01-15T14:33:42.294 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/163 2024-01-15T14:33:42.496 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/163 2024-01-15T14:33:42.659 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 135/163 2024-01-15T14:33:44.142 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T14:33:44.461 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T14:33:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T14:33:44.975 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 137/163 2024-01-15T14:33:45.134 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T14:33:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-15T14:33:45.335 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 138/163 2024-01-15T14:33:45.599 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 139/163 2024-01-15T14:33:45.622 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 137/163 2024-01-15T14:33:45.820 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 140/163 2024-01-15T14:33:45.940 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 138/163 2024-01-15T14:33:46.011 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 141/163 2024-01-15T14:33:46.162 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 139/163 2024-01-15T14:33:46.306 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 142/163 2024-01-15T14:33:46.324 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 140/163 2024-01-15T14:33:46.500 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 141/163 2024-01-15T14:33:46.579 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 143/163 2024-01-15T14:33:46.780 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 142/163 2024-01-15T14:33:46.802 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 144/163 2024-01-15T14:33:46.813 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 144/163 2024-01-15T14:33:47.023 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 145/163 2024-01-15T14:33:47.089 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 143/163 2024-01-15T14:33:47.219 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flex-2.6.1-9.el8.x86_64 146/163 2024-01-15T14:33:47.282 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/163 2024-01-15T14:33:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 146/163 2024-01-15T14:33:47.358 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/163 2024-01-15T14:33:47.543 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : bison-3.0.4-10.el8.x86_64 147/163 2024-01-15T14:33:47.556 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 147/163 2024-01-15T14:33:47.583 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/163 2024-01-15T14:33:47.596 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/163 2024-01-15T14:33:47.726 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 148/163 2024-01-15T14:33:47.758 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 146/163 2024-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 146/163 2024-01-15T14:33:47.902 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 147/163 2024-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 149/163 2024-01-15T14:33:48.047 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 148/163 2024-01-15T14:33:48.073 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/163 2024-01-15T14:33:48.073 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 149/163 2024-01-15T14:33:48.386 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/163 2024-01-15T14:33:48.386 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T14:33:48.388 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T14:33:48.389 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T14:33:48.614 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T14:33:48.614 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T14:33:48.652 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-15T14:33:48.652 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T14:33:48.837 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-15T14:33:48.837 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T14:33:48.875 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T14:33:49.064 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-15T14:33:49.120 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 154/163 2024-01-15T14:33:49.279 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 155/163 2024-01-15T14:33:49.309 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 154/163 2024-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 155/163 2024-01-15T14:33:49.529 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 156/163 2024-01-15T14:33:49.529 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T14:33:49.693 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 156/163 2024-01-15T14:33:49.693 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T14:33:49.788 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T14:33:49.788 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T14:33:49.910 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-15T14:33:49.910 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T14:33:50.046 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T14:33:50.047 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T14:33:50.118 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 158/163 2024-01-15T14:33:50.118 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T14:33:50.327 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-15T14:33:50.327 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T14:33:50.551 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T14:33:50.551 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T14:33:50.793 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T14:33:50.829 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-15T14:33:50.952 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 162/163 2024-01-15T14:33:50.952 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T14:33:51.004 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 162/163 2024-01-15T14:33:51.004 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 1/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 3/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 4/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 5/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 6/163 2024-01-15T14:33:51.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 7/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 8/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 14/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 15/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 16/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 17/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 18/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 19/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 20/163 2024-01-15T14:33:51.848 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 21/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 22/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 23/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 24/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 25/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 26/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 27/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 28/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 29/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 30/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 31/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 32/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 34/163 2024-01-15T14:33:51.849 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 35/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 36/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 37/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 38/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 42/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 43/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 44/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 45/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 46/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 47/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 48/163 2024-01-15T14:33:51.850 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 49/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 56/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 57/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 58/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/163 2024-01-15T14:33:51.851 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 71/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 72/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 73/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 75/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/163 2024-01-15T14:33:51.852 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 80/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 82/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 83/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 84/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 85/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 86/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 87/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 88/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 89/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 90/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 91/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 92/163 2024-01-15T14:33:51.853 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 93/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 94/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 95/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 96/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 97/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 98/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 99/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/163 2024-01-15T14:33:51.854 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 111/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 112/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 113/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 114/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 115/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 116/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 117/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 118/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 119/163 2024-01-15T14:33:51.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 120/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 121/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 122/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.1-8.el8.noarch 123/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpq-13.11-1.el8.x86_64 124/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-7.el8.x86_64 125/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 126/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 127/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 128/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 129/163 2024-01-15T14:33:51.856 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 130/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 131/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 132/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 133/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 134/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 135/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 136/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 137/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 138/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 139/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 140/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 141/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 142/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 143/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 144/163 2024-01-15T14:33:51.857 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 145/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 146/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 147/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 148/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 149/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 150/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 151/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 152/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 153/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 154/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 155/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 156/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 157/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 158/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 159/163 2024-01-15T14:33:51.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 160/163 2024-01-15T14:33:51.859 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/163 2024-01-15T14:33:51.859 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 162/163 2024-01-15T14:33:51.956 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 1/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 3/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 4/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 5/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 6/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 7/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 8/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 9/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 10/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 11/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 12/163 2024-01-15T14:33:51.957 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 13/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_ 14/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 15/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 16/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_6 17/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86 18/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 19/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 20/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g95c90f41.el8. 21/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_6 22/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 23/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 24/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 25/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 26/163 2024-01-15T14:33:51.958 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 27/163 2024-01-15T14:33:51.959 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 28/163 2024-01-15T14:33:51.959 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 29/163 2024-01-15T14:33:51.959 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noa 30/163 2024-01-15T14:33:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.n 31/163 2024-01-15T14:33:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 32/163 2024-01-15T14:33:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 33/163 2024-01-15T14:33:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 34/163 2024-01-15T14:33:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 35/163 2024-01-15T14:33:51.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 36/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 37/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 38/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 42/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 43/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 44/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 45/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 46/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 47/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 48/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 49/163 2024-01-15T14:33:51.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 56/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 57/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-2.6-1.el8.noarch 58/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/163 2024-01-15T14:33:51.973 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 71/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 72/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 73/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 74/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 75/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 76/163 2024-01-15T14:33:51.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 77/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 78/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 79/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flex-2.6.1-9.el8.x86_64 81/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 82/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 83/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 84/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 86/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 88/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 89/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 90/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 91/163 2024-01-15T14:33:51.975 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 92/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 93/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 94/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 95/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 96/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : automake-1.16.1-8.el8.noarch 97/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpq-13.11-1.el8.x86_64 98/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-7.el8.x86_64 99/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 100/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 101/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 102/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 103/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 104/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/163 2024-01-15T14:33:51.976 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 106/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 107/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 108/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 110/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 111/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 112/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 113/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 114/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 115/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 116/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 117/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 118/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 119/163 2024-01-15T14:33:51.977 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 120/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 121/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 122/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 123/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 124/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 125/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 126/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 127/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 128/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 129/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 131/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 132/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 133/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 134/163 2024-01-15T14:33:51.978 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 135/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 136/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 137/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 138/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 139/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 140/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 141/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 142/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 143/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 144/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 145/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 146/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 147/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 148/163 2024-01-15T14:33:51.979 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 149/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 150/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 151/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 152/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 153/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 154/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 155/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 156/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 157/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 158/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 159/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 160/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/163 2024-01-15T14:33:51.980 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 162/163 2024-01-15T14:33:58.838 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/163 2024-01-15T14:33:58.838 INFO:teuthology.orchestra.run.smithi159.stdout:Installed products updated. 2024-01-15T14:33:58.877 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/163 2024-01-15T14:33:58.877 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2024-01-15T14:33:59.449 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:33:59.449 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.450 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-15T14:33:59.451 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.1-8.el8.noarch 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.0.4-10.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.452 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.453 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.1-9.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-15T14:33:59.454 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-7.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-15T14:33:59.455 INFO:teuthology.orchestra.run.smithi077.stdout: libpq-13.11-1.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-15T14:33:59.456 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.5.0-1.el8.noarch 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-15T14:33:59.457 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-15T14:33:59.458 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-15T14:33:59.459 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-15T14:33:59.460 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-15T14:33:59.461 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-15T14:33:59.462 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-15T14:33:59.463 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-15T14:33:59.464 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-15T14:33:59.465 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-15T14:33:59.465 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:33:59.465 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-15T14:33:59.705 DEBUG:teuthology.parallel:result is None 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.935 INFO:teuthology.orchestra.run.smithi159.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: automake-1.16.1-8.el8.noarch 2024-01-15T14:33:59.936 INFO:teuthology.orchestra.run.smithi159.stdout: bison-3.0.4-10.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.937 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: cephfs-top-2:18.0.0-7629.g95c90f41.el8.noarch 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: flex-2.6.1-9.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-7.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-15T14:33:59.938 INFO:teuthology.orchestra.run.smithi159.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libpq-13.11-1.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-15T14:33:59.939 INFO:teuthology.orchestra.run.smithi159.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.5.0-1.el8.noarch 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-15T14:33:59.940 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-15T14:33:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-15T14:33:59.942 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-2.6-1.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-15T14:33:59.943 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.944 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:18.0.0-7629.g95c90f41.el8.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:33:59.945 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-01-15T14:34:01.325 DEBUG:teuthology.parallel:result is None 2024-01-15T14:34:01.326 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T14:34:01.468 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-15T14:34:01.497 INFO:teuthology.orchestra.run.smithi077.stdout:18.0.0-7629.g95c90f41.el8 2024-01-15T14:34:01.497 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g95c90f41.el8 2024-01-15T14:34:01.498 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g95c90f41 is installed. 2024-01-15T14:34:01.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T14:34:01.646 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-15T14:34:01.671 INFO:teuthology.orchestra.run.smithi159.stdout:18.0.0-7629.g95c90f41.el8 2024-01-15T14:34:01.671 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g95c90f41.el8 2024-01-15T14:34:01.671 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g95c90f41 is installed. 2024-01-15T14:34:01.672 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-15T14:34:01.673 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:01.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T14:34:01.722 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:01.722 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T14:34:01.764 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-15T14:34:01.765 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:01.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T14:34:01.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T14:34:01.864 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:01.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T14:34:01.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T14:34:01.960 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-15T14:34:01.960 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:01.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T14:34:01.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T14:34:02.056 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:02.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T14:34:02.084 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T14:34:02.151 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-15T14:34:02.151 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:02.151 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T14:34:02.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T14:34:02.247 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:02.247 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T14:34:02.273 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T14:34:02.340 INFO:teuthology.run_tasks:Running task ceph... 2024-01-15T14:34:02.414 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-15T14:34:02.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2024-01-15T14:34:02.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-01-15T14:34:02.443 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-15T14:34:02.444 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T14:34:02.484 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T14:34:02.511 INFO:tasks.ceph:Creating extra log directories... 2024-01-15T14:34:02.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T14:34:02.551 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T14:34:02.584 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-15T14:34:02.584 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-15T14:34:02.584 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-15T14:34:02.584 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:02.585 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T14:34:02.632 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T14:34:02.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-15T14:34:02.773 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:02.773 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T14:34:02.790 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T14:34:02.903 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-15T14:34:02.935 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-15T14:34:02.935 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-15T14:34:02.935 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7517082', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-vol-clone-stats-temp', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2171, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'mgr-vol-clone-stats-temp', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAm4BXul+XHUCskQiKefJarpQ9/RKYyRFJFCclOpgH06y89Jd2ZojDz9DQif+pbcTJGbWq9qELPAYP9m+tbmoHw=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSr5y6GfvbWAbi7CR/OsML6+UDb2zXubrvEUEnenhEBJeiHQSGr42UrAhK7RFT/ydM05oOJJLic2pNGBqryQ5g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_14:18:41', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-15T14:34:02.935 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T14:34:02.937 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T14:34:02.987 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T14:34:02.994 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T14:34:03.055 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:03.055 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T14:34:03.075 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T14:34:03.076 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 174622 Links: 1 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-15 14:33:51.478814450 +0000 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-15 14:30:53.827963813 +0000 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-15 14:30:53.827963813 +0000 2024-01-15T14:34:03.133 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-15T14:34:03.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T14:34:03.197 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-15T14:34:03.197 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-15T14:34:03.198 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000183714 s, 2.8 MB/s 2024-01-15T14:34:03.199 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T14:34:03.257 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 172917 Links: 1 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-15 14:33:51.478814450 +0000 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-15 14:30:54.164956055 +0000 2024-01-15T14:34:03.314 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-15 14:30:54.164956055 +0000 2024-01-15T14:34:03.315 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-15T14:34:03.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000286883 s, 1.8 MB/s 2024-01-15T14:34:03.381 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T14:34:03.439 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-01-15T14:34:03.497 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T14:34:03.497 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:03.497 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 176369 Links: 1 2024-01-15T14:34:03.498 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:03.498 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:03.498 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-15 14:33:51.478814450 +0000 2024-01-15T14:34:03.498 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-15 14:30:54.511948066 +0000 2024-01-15T14:34:03.498 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-15 14:30:54.511948066 +0000 2024-01-15T14:34:03.498 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-15T14:34:03.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T14:34:03.563 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-15T14:34:03.563 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-15T14:34:03.563 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000183687 s, 2.8 MB/s 2024-01-15T14:34:03.564 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T14:34:03.622 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-01-15T14:34:03.680 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T14:34:03.680 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:03.680 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 177672 Links: 1 2024-01-15T14:34:03.680 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:03.680 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:03.681 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-15 14:33:51.478814450 +0000 2024-01-15T14:34:03.681 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-15 14:30:54.844940400 +0000 2024-01-15T14:34:03.681 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-15 14:30:54.844940400 +0000 2024-01-15T14:34:03.681 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-15T14:34:03.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T14:34:03.746 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-15T14:34:03.746 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-15T14:34:03.746 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00022487 s, 2.3 MB/s 2024-01-15T14:34:03.747 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T14:34:03.806 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:03.806 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:03.806 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T14:34:03.825 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T14:34:03.825 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 178181 Links: 1 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-01-15 14:33:51.597098219 +0000 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-01-15 14:30:44.624568704 +0000 2024-01-15T14:34:03.884 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-01-15 14:30:44.624568704 +0000 2024-01-15T14:34:03.885 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-01-15T14:34:03.885 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T14:34:03.951 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-01-15T14:34:03.951 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-01-15T14:34:03.952 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000248995 s, 2.1 MB/s 2024-01-15T14:34:03.953 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T14:34:04.012 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 174542 Links: 1 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-01-15 14:33:51.597098219 +0000 2024-01-15T14:34:04.072 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-01-15 14:30:44.964562180 +0000 2024-01-15T14:34:04.073 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-01-15 14:30:44.964562180 +0000 2024-01-15T14:34:04.073 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-01-15T14:34:04.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T14:34:04.138 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-01-15T14:34:04.138 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-01-15T14:34:04.139 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000244342 s, 2.1 MB/s 2024-01-15T14:34:04.140 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T14:34:04.198 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 174599 Links: 1 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-01-15 14:33:51.597098219 +0000 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-01-15 14:30:45.313555483 +0000 2024-01-15T14:34:04.257 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-01-15 14:30:45.313555483 +0000 2024-01-15T14:34:04.258 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-01-15T14:34:04.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T14:34:04.323 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-01-15T14:34:04.323 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-01-15T14:34:04.323 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000187053 s, 2.7 MB/s 2024-01-15T14:34:04.324 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T14:34:04.382 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-01-15T14:34:04.439 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 174674 Links: 1 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-01-15 14:33:51.597098219 +0000 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-01-15 14:30:45.656548901 +0000 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-01-15 14:30:45.656548901 +0000 2024-01-15T14:34:04.440 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-01-15T14:34:04.441 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T14:34:04.506 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-01-15T14:34:04.506 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-01-15T14:34:04.506 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000244077 s, 2.1 MB/s 2024-01-15T14:34:04.507 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T14:34:04.565 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:04.565 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T14:34:04.565 INFO:tasks.ceph:Generating config... 2024-01-15T14:34:04.566 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-15T14:34:04.567 INFO:tasks.ceph:[client] debug client = 20 2024-01-15T14:34:04.567 INFO:tasks.ceph:[client] debug ms = 1 2024-01-15T14:34:04.567 INFO:tasks.ceph:[client] pid file = 2024-01-15T14:34:04.567 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-15T14:34:04.567 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-15T14:34:04.567 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-01-15T14:34:04.567 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-15T14:34:04.567 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-15T14:34:04.567 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-15T14:34:04.567 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-15T14:34:04.568 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-15T14:34:04.569 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-15T14:34:04.569 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-15T14:34:04.569 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-15T14:34:04.569 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-15T14:34:04.569 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-15T14:34:04.569 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-15T14:34:04.569 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-15T14:34:04.569 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-15T14:34:04.569 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-15T14:34:04.569 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-15T14:34:04.570 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-15T14:34:04.571 INFO:tasks.ceph:Setting up mon.a... 2024-01-15T14:34:04.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-15T14:34:04.630 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2024-01-15T14:34:04.634 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-15T14:34:04.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T14:34:04.710 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77'), ('mon.b', '172.21.15.159'), ('mon.c', '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]')] 2024-01-15T14:34:04.710 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-15T14:34:04.711 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:04.711 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T14:34:04.767 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --add b 172.21.15.159 --addv c '[v2:172.21.15.159:3301,v1:172.21.15.159:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:04.856 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid a5eae108-559b-4366-a994-4d2898914163 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = pacific 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:fsid a5eae108-559b-4366-a994-4d2898914163 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2024-01-15T14:34:04.856992+0000 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-01-15T14:34:04.856992+0000 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 16 (pacific) 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-01-15T14:34:04.857 INFO:teuthology.orchestra.run.smithi077.stdout:1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.b 2024-01-15T14:34:04.858 INFO:teuthology.orchestra.run.smithi077.stdout:2: [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] mon.c 2024-01-15T14:34:04.858 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-15T14:34:04.859 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T14:34:04.876 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a5eae108-559b-4366-a994-4d2898914163... 2024-01-15T14:34:04.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T14:34:04.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.964 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-01-15T14:34:04.965 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.966 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on skipped message = False 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = a5eae108-559b-4366-a994-4d2898914163 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-01-15T14:34:04.967 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-15T14:34:04.971 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: debug finisher = 20 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.77,172.21.15.159,[v2:172.21.15.159:3301,v1:172.21.15.159:6790]" 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on skipped message = False 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = a5eae108-559b-4366-a994-4d2898914163 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: bdev async discard = True 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: bdev enable discard = True 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore allocator = bitmap 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 1/20 2024-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 1/20 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 4/10 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug finisher = 20 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: mon op complaint time = 120 2024-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: client mount timeout = 600 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 20 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = "" 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 900 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 900 2024-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout:[mds] 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 20 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds balancer = 20 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug frag = True 2024-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi159.stdout: mds debug scatterstat = True 2024-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi159.stdout: mds op complaint time = 180 2024-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi159.stdout: mds verify scatter = True 2024-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi159.stdout: osd op complaint time = 180 2024-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi159.stdout: rados mon op timeout = 900 2024-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi159.stdout: rados osd op timeout = 900 2024-01-15T14:34:04.993 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-15T14:34:04.994 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-15T14:34:05.090 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-15T14:34:05.090 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:05.091 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-15T14:34:05.108 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:05.108 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-15T14:34:05.165 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:34:05.165 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:05.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T14:34:05.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T14:34:05.241 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:05.242 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:05.299 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:34:05.299 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:05.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T14:34:05.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T14:34:05.339 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:05.339 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:05.397 INFO:tasks.ceph:Setting up mon nodes... 2024-01-15T14:34:05.397 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-15T14:34:05.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T14:34:05.451 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T14:34:05.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T14:34:05.546 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T14:34:05.549 INFO:tasks.ceph:Setting up mds nodes... 2024-01-15T14:34:05.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T14:34:05.604 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T14:34:05.607 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-15T14:34:05.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T14:34:05.730 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-15T14:34:05.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T14:34:05.812 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T14:34:05.815 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-15T14:34:05.842 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T14:34:05.936 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T14:34:05.939 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-15T14:34:05.966 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-15T14:34:05.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-15T14:34:06.010 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-15T14:34:06.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-15T14:34:06.067 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-15T14:34:06.080 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-15T14:34:06.137 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-15T14:34:06.182 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-15T14:34:06.195 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-15T14:34:06.195 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T14:34:06.195 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-15T14:34:06.222 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:06.223 INFO:tasks.ceph:role: osd.0 2024-01-15T14:34:06.223 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:34:06.223 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T14:34:06.296 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:06.297 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:06.727 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-15T14:34:06.729 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-15T14:34:06.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-15T14:34:06.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-15T14:34:06.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-15T14:34:07.035 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:07.035 INFO:tasks.ceph:role: osd.1 2024-01-15T14:34:07.035 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:34:07.035 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T14:34:07.113 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:07.114 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:07.528 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-15T14:34:07.531 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-15T14:34:07.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-15T14:34:07.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-15T14:34:07.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-15T14:34:07.744 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:07.744 INFO:tasks.ceph:role: osd.2 2024-01-15T14:34:07.744 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:34:07.744 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T14:34:07.819 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:07.819 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:07.819 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:07.819 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:07.819 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:07.820 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:07.820 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:07.820 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:07.820 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:07.820 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:08.261 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-15T14:34:08.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-15T14:34:08.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-15T14:34:08.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-15T14:34:08.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-15T14:34:08.559 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:08.559 INFO:tasks.ceph:role: osd.3 2024-01-15T14:34:08.559 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:34:08.559 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:08.636 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:09.069 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-15T14:34:09.071 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-15T14:34:09.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-15T14:34:09.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-15T14:34:09.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:09.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:09.364+0000 7f166771a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-15T14:34:09.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:09.364+0000 7f166771a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-15T14:34:09.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:09.364+0000 7f166771a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T14:34:09.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:09.364+0000 7f166771a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T14:34:09.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:09.364+0000 7f166771a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T14:34:09.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:09.365+0000 7f166771a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-15T14:34:12.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-15T14:34:12.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:12.340+0000 7f014273b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-15T14:34:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:12.340+0000 7f014273b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-15T14:34:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:12.340+0000 7f014273b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T14:34:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:12.340+0000 7f014273b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T14:34:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:12.340+0000 7f014273b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T14:34:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:12.341+0000 7f014273b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-15T14:34:15.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-15T14:34:15.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:15.322+0000 7f6d6cf00640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-15T14:34:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:15.322+0000 7f6d6cf00640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-15T14:34:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:15.322+0000 7f6d6cf00640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T14:34:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:15.322+0000 7f6d6cf00640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T14:34:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:15.322+0000 7f6d6cf00640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T14:34:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:15.323+0000 7f6d6cf00640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-15T14:34:18.182 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-15T14:34:18.214 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:18.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:18.308+0000 7f86b0044640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-15T14:34:18.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:18.308+0000 7f86b0044640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-15T14:34:18.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:18.308+0000 7f86b0044640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T14:34:18.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:18.308+0000 7f86b0044640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T14:34:18.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:18.308+0000 7f86b0044640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T14:34:18.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:18.309+0000 7f86b0044640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-15T14:34:21.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-15T14:34:21.199 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-15T14:34:21.232 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:21.232 INFO:tasks.ceph:role: osd.4 2024-01-15T14:34:21.233 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:34:21.233 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:21.311 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:21.312 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-01-15T14:34:21.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-01-15T14:34:21.751 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-15T14:34:21.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-15T14:34:21.971 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-15T14:34:22.040 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:22.040 INFO:tasks.ceph:role: osd.5 2024-01-15T14:34:22.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:34:22.040 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T14:34:22.115 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:22.116 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:22.535 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-01-15T14:34:22.537 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-01-15T14:34:22.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-15T14:34:22.652 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-15T14:34:22.686 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-15T14:34:22.755 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:22.755 INFO:tasks.ceph:role: osd.6 2024-01-15T14:34:22.755 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:34:22.755 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T14:34:22.831 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:22.831 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:22.831 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:22.831 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:22.831 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:22.831 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:22.832 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:22.832 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:22.832 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:22.832 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:23.280 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-01-15T14:34:23.282 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-01-15T14:34:23.282 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-15T14:34:23.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-15T14:34:23.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-15T14:34:23.568 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T14:34:23.568 INFO:tasks.ceph:role: osd.7 2024-01-15T14:34:23.568 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:34:23.568 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T14:34:23.646 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T14:34:23.646 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T14:34:23.646 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T14:34:23.646 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T14:34:23.647 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T14:34:23.647 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-01-15T14:34:23.647 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T14:34:23.647 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T14:34:23.647 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T14:34:23.647 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T14:34:24.088 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-01-15T14:34:24.090 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-01-15T14:34:24.090 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-15T14:34:24.256 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-15T14:34:24.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:24.383 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:24.382+0000 7fd76ddbe640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-15T14:34:24.383 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:24.383+0000 7fd76ddbe640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-15T14:34:24.383 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:24.383+0000 7fd76ddbe640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T14:34:24.383 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:24.383+0000 7fd76ddbe640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T14:34:24.383 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:24.383+0000 7fd76ddbe640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T14:34:24.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:24.383+0000 7fd76ddbe640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-15T14:34:27.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-15T14:34:27.268 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:27.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:27.360+0000 7fa7c68f5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-15T14:34:27.361 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:27.360+0000 7fa7c68f5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-15T14:34:27.361 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:27.360+0000 7fa7c68f5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T14:34:27.361 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:27.360+0000 7fa7c68f5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T14:34:27.361 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:27.360+0000 7fa7c68f5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T14:34:27.361 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:27.360+0000 7fa7c68f5640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-15T14:34:30.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-15T14:34:30.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:30.328 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:30.327+0000 7ff6b47b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-15T14:34:30.328 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:30.328+0000 7ff6b47b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-15T14:34:30.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:30.328+0000 7ff6b47b2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T14:34:30.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:30.328+0000 7ff6b47b2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T14:34:30.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:30.328+0000 7ff6b47b2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T14:34:30.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:30.328+0000 7ff6b47b2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-15T14:34:32.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:34:32.940 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:34:33.186 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-15T14:34:33.216 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:33.307 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:33.307+0000 7f768da91640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-15T14:34:33.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:33.307+0000 7f768da91640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-15T14:34:33.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:33.307+0000 7f768da91640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T14:34:33.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:33.307+0000 7f768da91640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T14:34:33.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:33.307+0000 7f768da91640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T14:34:33.308 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:33.308+0000 7f768da91640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-15T14:34:36.164 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-15T14:34:36.196 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-15T14:34:36.196 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-15T14:34:36.229 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-15T14:34:36.299 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-15T14:34:36.370 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-15T14:34:36.439 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-15T14:34:36.509 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-15T14:34:36.578 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:36.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-15T14:34:36.647 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:36.648 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-15T14:34:36.680 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:36.681 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-15T14:34:36.752 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:36.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-15T14:34:36.823 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:36.823 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-15T14:34:36.892 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:36.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-15T14:34:36.962 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:36.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-15T14:34:37.030 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:37.030 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-15T14:34:37.097 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:37.097 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-15T14:34:37.117 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:37.117 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-15T14:34:37.176 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:37.176 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-15T14:34:37.194 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:37.194 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-15T14:34:37.251 INFO:tasks.ceph:Adding keys to all mons... 2024-01-15T14:34:37.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T14:34:37.255 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.y] 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdQqVlY0v0GhAAua4VooMS6v1Lii5ufEofbg== 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdQqVljxYOJBAA6tNJno9y9x+zGr1FiZN2hA== 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdQqVlGw9oKxAArSX/vvELUO3UU9nSNCmURg== 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2024-01-15T14:34:37.296 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBhQqVlkqHPFRAAJ/qFoPn4R8+VN6fBlM5kFg== 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBkQqVlWohgFBAAm6nn3wcmSfWKALwZ63ZR/g== 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBnQqVlIvlKExAAXXy2r2cr61BvOivASiwypQ== 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBqQqVlfiVlEhAAXi3wsvTmPRGeBGx7e6pD2w== 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdQqVlHpecIBAAxACo580Z7g9U4IHsV17hXQ== 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdQqVl+5FzMBAAra8qc3O5AH9FOYwASZIuag== 2024-01-15T14:34:37.297 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdQqVl7m/ZNxAA41CCPNutEM+PKAaLqzBYng== 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.4] 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBwQqVli8LcFhAARSvQ90Fmunp2XMM2r4/BBQ== 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.5] 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBzQqVlnfmBFRAAUEgbCE4QFeX4xrgIf0rLng== 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.6] 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQB2QqVlOi6XExAA/J69KO/nyj6ggyGu6Tnx3Q== 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.7] 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQB5QqVlMFldEhAA7mY2tYGH8cvWIBxqaNqBuw== 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-01-15T14:34:37.298 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBeQqVl3gmtABAA2JDAPlTu6jT/iTUZPZjfVw== 2024-01-15T14:34:37.299 INFO:teuthology.orchestra.run.smithi077.stdout:[client.2] 2024-01-15T14:34:37.299 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBeQqVlfSMMBBAACXaauvbUylsDhE6gLtK4jw== 2024-01-15T14:34:37.299 INFO:teuthology.orchestra.run.smithi077.stdout:[client.1] 2024-01-15T14:34:37.299 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBeQqVlrWl0BxAAQNlFMK3OnPmkHN0sasy5Kw== 2024-01-15T14:34:37.299 INFO:teuthology.orchestra.run.smithi077.stdout:[client.3] 2024-01-15T14:34:37.299 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBeQqVlcqDeChAAe/SEZKUY/KknjkYmcbuj+w== 2024-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.y] 2024-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBdQqVlY0v0GhAAua4VooMS6v1Lii5ufEofbg== 2024-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.a] 2024-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBdQqVljxYOJBAA6tNJno9y9x+zGr1FiZN2hA== 2024-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.c] 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBdQqVlGw9oKxAArSX/vvELUO3UU9nSNCmURg== 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBhQqVlkqHPFRAAJ/qFoPn4R8+VN6fBlM5kFg== 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBkQqVlWohgFBAAm6nn3wcmSfWKALwZ63ZR/g== 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBnQqVlIvlKExAAXXy2r2cr61BvOivASiwypQ== 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBqQqVlfiVlEhAAXi3wsvTmPRGeBGx7e6pD2w== 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBdQqVlHpecIBAAxACo580Z7g9U4IHsV17hXQ== 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.b] 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBdQqVl+5FzMBAAra8qc3O5AH9FOYwASZIuag== 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout:[mds.d] 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBdQqVl7m/ZNxAA41CCPNutEM+PKAaLqzBYng== 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.4] 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBwQqVli8LcFhAARSvQ90Fmunp2XMM2r4/BBQ== 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.5] 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBzQqVlnfmBFRAAUEgbCE4QFeX4xrgIf0rLng== 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.6] 2024-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQB2QqVlOi6XExAA/J69KO/nyj6ggyGu6Tnx3Q== 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.7] 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQB5QqVlMFldEhAA7mY2tYGH8cvWIBxqaNqBuw== 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBeQqVl3gmtABAA2JDAPlTu6jT/iTUZPZjfVw== 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout:[client.2] 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBeQqVlfSMMBBAACXaauvbUylsDhE6gLtK4jw== 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout:[client.1] 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBeQqVlrWl0BxAAQNlFMK3OnPmkHN0sasy5Kw== 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout:[client.3] 2024-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBeQqVlcqDeChAAe/SEZKUY/KknjkYmcbuj+w== 2024-01-15T14:34:37.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T14:34:37.344 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T14:34:37.417 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.480 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.540 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.597 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.653 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T14:34:37.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T14:34:37.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.823 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.882 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T14:34:37.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.941 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:37.998 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:38.052 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:38.055 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:38.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:38.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T14:34:38.167 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.170 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.227 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T14:34:38.408 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-15T14:34:38.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-15T14:34:38.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T14:34:38.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-15T14:34:38.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-15T14:34:38.588 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T14:34:38.688 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-15T14:34:38.714 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-15T14:34:38.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T14:34:38.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-15T14:34:38.910 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:38.913 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T14:34:38.968 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-15T14:34:38.968 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-15T14:34:38.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-15T14:34:38.971 INFO:tasks.ceph.mon.a:Started 2024-01-15T14:34:38.971 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-15T14:34:38.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-15T14:34:39.011 INFO:tasks.ceph.mon.b:Started 2024-01-15T14:34:39.011 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-15T14:34:39.011 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-15T14:34:39.013 INFO:tasks.ceph.mon.c:Started 2024-01-15T14:34:39.013 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-15T14:34:39.014 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-15T14:34:39.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-15T14:34:39.017 INFO:tasks.ceph.mgr.y:Started 2024-01-15T14:34:39.017 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-15T14:34:39.017 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-15T14:34:39.020 INFO:tasks.ceph.mgr.x:Started 2024-01-15T14:34:39.020 INFO:tasks.ceph:Setting crush tunables to default 2024-01-15T14:34:39.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2024-01-15T14:34:39.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.143+0000 7fd61a0ae700 1 Processor -- start 2024-01-15T14:34:39.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.143+0000 7fd61a0ae700 1 -- start start 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd61a0ae700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd61a0ae700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd61a0ae700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd61a0ae700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61406dc60 con 0x7fd61406e990 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd61a0ae700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61406e500 con 0x7fd614073f90 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd61a0ae700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6140748b0 con 0x7fd6140addf0 2024-01-15T14:34:39.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37136/0 (socket says 172.21.15.77:37136) 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 -- 172.21.15.77:0/3785502488 learned_addr learned my addr 172.21.15.77:0/3785502488 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd6137fe700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 -- 172.21.15.77:0/3785502488 >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 msgr2=0x7fd61406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 -- 172.21.15.77:0/3785502488 >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 msgr2=0x7fd61406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd612ffd700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd613fff700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd6137fe700 1 -- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 msgr2=0x7fd614074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-15T14:34:39.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd6137fe700 1 -- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 msgr2=0x7fd614074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T14:34:39.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd6137fe700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T14:34:39.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd612ffd700 1 -- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 msgr2=0x7fd6140cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-15T14:34:39.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd612ffd700 1 -- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 msgr2=0x7fd6140cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T14:34:39.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd6137fe700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T14:34:39.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd612ffd700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T14:34:39.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.144+0000 7fd612ffd700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.77:3300/0 conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd612ffd700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd6137fe700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 -- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 msgr2=0x7fd6140cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 -- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 msgr2=0x7fd614074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 -- 172.21.15.77:0/3785502488 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6140cd7b0 con 0x7fd61406e990 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd612ffd700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd613fff700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd61406e990 0x7fd61406ee20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6140736e0 tx=0x7fd604009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57745f695477f14d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd6127fc700 1 -- 172.21.15.77:0/3785502488 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd604004230 con 0x7fd61406e990 2024-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd6127fc700 1 -- 172.21.15.77:0/3785502488 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd604004390 con 0x7fd61406e990 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.345+0000 7fd6127fc700 1 -- 172.21.15.77:0/3785502488 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6040038e0 con 0x7fd61406e990 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3785502488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd61406e990 msgr2=0x7fd61406ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd61406e990 0x7fd61406ee20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6140736e0 tx=0x7fd604009c30 comp rx=0 tx=0).stop 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3785502488 shutdown_connections 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3785502488 >> v2:172.21.15.159:3300/0 conn(0x7fd614073f90 0x7fd614074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd61406e990 0x7fd61406ee20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3785502488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd6140cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3785502488 >> 172.21.15.77:0/3785502488 conn(0x7fd614072b20 msgr2=0x7fd61406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3785502488 shutdown_connections 2024-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.346+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3785502488 wait complete. 2024-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.347+0000 7fd61a0ae700 1 Processor -- start 2024-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.347+0000 7fd61a0ae700 1 -- start start 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd61a0ae700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd61406e990 0x7fd61415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd61a0ae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 0x7fd6141f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd61a0ae700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd614156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd61a0ae700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6141f5990 con 0x7fd614073f90 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd61a0ae700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd614157040 con 0x7fd61406e990 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd61a0ae700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6141f50f0 con 0x7fd6140addf0 2024-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd612ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 0x7fd6141f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd612ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 0x7fd6141f4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35470/0 (socket says 172.21.15.77:35470) 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd612ffd700 1 -- 172.21.15.77:0/3969674931 learned_addr learned my addr 172.21.15.77:0/3969674931 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd613fff700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd614156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.348+0000 7fd6137fe700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd61406e990 0x7fd61415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd613fff700 1 -- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd61406e990 msgr2=0x7fd61415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd613fff700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd61406e990 0x7fd61415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd613fff700 1 -- 172.21.15.77:0/3969674931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 msgr2=0x7fd6141f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd613fff700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 0x7fd6141f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd613fff700 1 -- 172.21.15.77:0/3969674931 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6040095b0 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd612ffd700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 0x7fd6141f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd613fff700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd614156b00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd604017000 tx=0x7fd6040047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd604027430 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd604022070 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6141f53d0 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6040277f0 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61409b030 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd604024030 con 0x7fd6140addf0 2024-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.349+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd6040308f0 con 0x7fd6140addf0 2024-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.350+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61409ae70 con 0x7fd6140addf0 2024-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.353+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd60400c070 con 0x7fd6140addf0 2024-01-15T14:34:39.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.469+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd614069020 con 0x7fd6140addf0 2024-01-15T14:34:39.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.477+0000 7fd60bfff700 1 -- 172.21.15.77:0/3969674931 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd60400f040 con 0x7fd6140addf0 2024-01-15T14:34:39.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 msgr2=0x7fd614156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:39.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd614156b00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd604017000 tx=0x7fd6040047c0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 shutdown_connections 2024-01-15T14:34:39.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd6140addf0 0x7fd614156b00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd61406e990 0x7fd61415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 --2- 172.21.15.77:0/3969674931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd614073f90 0x7fd6141f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:39.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 >> 172.21.15.77:0/3969674931 conn(0x7fd614072b20 msgr2=0x7fd6140b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:39.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 shutdown_connections 2024-01-15T14:34:39.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:39.478+0000 7fd61a0ae700 1 -- 172.21.15.77:0/3969674931 wait complete. 2024-01-15T14:34:39.481 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-01-15T14:34:39.492 INFO:tasks.ceph:check_enable_crimson: False 2024-01-15T14:34:39.492 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-15T14:34:39.492 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:39.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-15T14:34:39.518 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:39.519 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-15T14:34:39.585 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:39.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-15T14:34:39.653 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:34:39.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-15T14:34:39.719 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:39.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-15T14:34:39.748 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:39.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-15T14:34:39.800 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:39.800+0000 7fec277c6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T14:34:39.816 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:39.817 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-15T14:34:39.880 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:39.879+0000 7fec277c6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T14:34:39.883 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:34:39.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-15T14:34:39.950 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new d32a1a3d-1302-4ded-ba06-c91ff40bf18a 0 2024-01-15T14:34:39.953 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:39.952+0000 7fec277c6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T14:34:40.023 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.023+0000 7fec277c6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T14:34:40.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.112+0000 7f88f6d9f700 1 Processor -- start 2024-01-15T14:34:40.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.113+0000 7f88f6d9f700 1 -- start start 2024-01-15T14:34:40.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.113+0000 7f88f6d9f700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f88f00ae280 0x7f88f00ae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.113+0000 7f88f6d9f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f88f00b67a0 0x7f88f006dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.113+0000 7f88f6d9f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f0074be0 con 0x7f88f006eec0 2024-01-15T14:34:40.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.113+0000 7f88f6d9f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f015f1b0 con 0x7f88f00b67a0 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.113+0000 7f88f6d9f700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00aeba0 con 0x7f88f00ae280 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f88f00ae280 0x7f88f00ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f88f00ae280 0x7f88f00ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37036/0 (socket says 172.21.15.159:37036) 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f559c700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f88f006eec0 0x7f88f00b6090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:40814/0 (socket says 172.21.15.159:40814) 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 -- 172.21.15.159:0/3152617785 learned_addr learned my addr 172.21.15.159:0/3152617785 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f659e700 1 --2- 172.21.15.159:0/3152617785 >> v2:172.21.15.77:3300/0 conn(0x7f88f00b67a0 0x7f88f006dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 -- 172.21.15.159:0/3152617785 >> v2:172.21.15.77:3300/0 conn(0x7f88f00b67a0 msgr2=0x7f88f006dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 --2- 172.21.15.159:0/3152617785 >> v2:172.21.15.77:3300/0 conn(0x7f88f00b67a0 0x7f88f006dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 -- 172.21.15.159:0/3152617785 >> v1:172.21.15.159:6789/0 conn(0x7f88f006eec0 legacy=0x7f88f00b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 -- 172.21.15.159:0/3152617785 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f015e530 con 0x7f88f00ae280 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f4d9b700 1 -- 172.21.15.159:0/3152617785 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148278519 0 0) 0x7f88f0074be0 con 0x7f88f006eec0 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f659e700 1 --2- 172.21.15.159:0/3152617785 >> v2:172.21.15.77:3300/0 conn(0x7f88f00b67a0 0x7f88f006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f5d9d700 1 --2- 172.21.15.159:0/3152617785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f00ae660 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88f004f8e0 tx=0x7f88ec009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56788b582e27f061 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f4d9b700 1 -- 172.21.15.159:0/3152617785 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00b870 con 0x7f88f00ae280 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f4d9b700 1 -- 172.21.15.159:0/3152617785 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec00b9d0 con 0x7f88f00ae280 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.114+0000 7f88f4d9b700 1 -- 172.21.15.159:0/3152617785 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00bd60 con 0x7f88f00ae280 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 -- 172.21.15.159:0/3152617785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 msgr2=0x7f88f00ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/3152617785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f00ae660 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88f004f8e0 tx=0x7f88ec009bd0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 -- 172.21.15.159:0/3152617785 shutdown_connections 2024-01-15T14:34:40.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/3152617785 >> v2:172.21.15.77:3300/0 conn(0x7f88f00b67a0 0x7f88f006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/3152617785 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f00ae660 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 -- 172.21.15.159:0/3152617785 >> 172.21.15.159:0/3152617785 conn(0x7f88f00727f0 msgr2=0x7f88f006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 -- 172.21.15.159:0/3152617785 shutdown_connections 2024-01-15T14:34:40.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.115+0000 7f88f6d9f700 1 -- 172.21.15.159:0/3152617785 wait complete. 2024-01-15T14:34:40.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.117+0000 7f88f6d9f700 1 Processor -- start 2024-01-15T14:34:40.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.117+0000 7f88f6d9f700 1 -- start start 2024-01-15T14:34:40.123 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88f006eec0 0x7f88f0141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.123 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88f00b67a0 0x7f88f01558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0158200 con 0x7f88f006eec0 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f01572e0 con 0x7f88f00ae280 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0157580 con 0x7f88f00b67a0 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f559c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f559c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f01553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37042/0 (socket says 172.21.15.159:37042) 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f559c700 1 -- 172.21.15.159:0/1415374394 learned_addr learned my addr 172.21.15.159:0/1415374394 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88f00b67a0 0x7f88f01558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.124 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f5d9d700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88f006eec0 0x7f88f0141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 -- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 msgr2=0x7f88f01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f01553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 -- 172.21.15.159:0/1415374394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88f006eec0 msgr2=0x7f88f0141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88f006eec0 0x7f88f0141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 -- 172.21.15.159:0/1415374394 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec009550 con 0x7f88f00b67a0 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f559c700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f659e700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88f00b67a0 0x7f88f01558e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88e8007e60 tx=0x7f88e800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e800dd20 con 0x7f88f00b67a0 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e800f040 con 0x7f88f00b67a0 2024-01-15T14:34:40.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8003c60 con 0x7f88f00b67a0 2024-01-15T14:34:40.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f0157860 con 0x7f88f00b67a0 2024-01-15T14:34:40.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.118+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f0157e20 con 0x7f88f00b67a0 2024-01-15T14:34:40.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.119+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f88e8004330 con 0x7f88f00b67a0 2024-01-15T14:34:40.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.119+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f88e801fa30 con 0x7f88f00b67a0 2024-01-15T14:34:40.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.119+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f0158340 con 0x7f88f00b67a0 2024-01-15T14:34:40.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.121+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f88f0158340 con 0x7f88f00b67a0 2024-01-15T14:34:40.135 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.134+0000 7f95e115d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T14:34:40.184 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.183+0000 7fec277c6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T14:34:40.206 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.206+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d32a1a3d-1302-4ded-ba06-c91ff40bf18a", "id": 0} v 0) v1 -- 0x7f88f0068e70 con 0x7f88f00b67a0 2024-01-15T14:34:40.210 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.210+0000 7f88e67fc700 1 -- 172.21.15.159:0/1415374394 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d32a1a3d-1302-4ded-ba06-c91ff40bf18a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f88e801a070 con 0x7f88f00b67a0 2024-01-15T14:34:40.211 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-01-15T14:34:40.211 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.210+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88f00b67a0 msgr2=0x7f88f01558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.210+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88f00b67a0 0x7f88f01558e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88e8007e60 tx=0x7f88e800d3b0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 shutdown_connections 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f88f00b67a0 0x7f88f01558e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88f006eec0 0x7f88f0141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 --2- 172.21.15.159:0/1415374394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f88f00ae280 0x7f88f01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 >> 172.21.15.159:0/1415374394 conn(0x7f88f00727f0 msgr2=0x7f88f00b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 shutdown_connections 2024-01-15T14:34:40.212 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.211+0000 7f88f6d9f700 1 -- 172.21.15.159:0/1415374394 wait complete. 2024-01-15T14:34:40.218 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.218+0000 7f95e115d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T14:34:40.223 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 4079025f-8ebd-4dc4-a2b2-146beab483f2 1 2024-01-15T14:34:40.297 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.297+0000 7f95e115d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T14:34:40.338 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.337+0000 7fec277c6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T14:34:40.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.346+0000 7f7af2845700 1 Processor -- start 2024-01-15T14:34:40.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.346+0000 7f7af2845700 1 -- start start 2024-01-15T14:34:40.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af2845700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec071420 0x7f7aec071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af2845700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec071db0 con 0x7f7aec15a3e0 2024-01-15T14:34:40.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af2845700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec0b4750 con 0x7f7aec06eba0 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af2845700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec15a7c0 con 0x7f7aec071420 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af0de2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec071420 0x7f7aec071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7aeb7fe700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7aec06eba0 0x7f7aec0ce070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:40818/0 (socket says 172.21.15.159:40818) 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7aeb7fe700 1 -- 172.21.15.159:0/2353080594 learned_addr learned my addr 172.21.15.159:0/2353080594 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7aeaffd700 1 -- 172.21.15.159:0/2353080594 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816370298 0 0) 0x7f7aec0b4750 con 0x7f7aec06eba0 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7aeaffd700 1 -- 172.21.15.159:0/2353080594 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad40046c0 con 0x7f7aec06eba0 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af0de2700 1 -- 172.21.15.159:0/2353080594 >> v1:172.21.15.159:6789/0 conn(0x7f7aec06eba0 legacy=0x7f7aec0ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af0de2700 1 -- 172.21.15.159:0/2353080594 >> v1:172.21.15.77:6789/0 conn(0x7f7aec15a3e0 legacy=0x7f7aec06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7aeaffd700 1 -- 172.21.15.159:0/2353080594 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794549438 0 0) 0x7f7adc001110 con 0x7f7aec15a3e0 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af0de2700 1 -- 172.21.15.159:0/2353080594 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0cf9f0 con 0x7f7aec071420 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.348+0000 7f7af0de2700 1 --2- 172.21.15.159:0/2353080594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec071420 0x7f7aec071870 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009dc0 tx=0x7f7ae400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b48e6c8312b52d43 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.349+0000 7f7aeaffd700 1 -- 172.21.15.159:0/2353080594 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400e070 con 0x7f7aec071420 2024-01-15T14:34:40.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.349+0000 7f7aeaffd700 1 -- 172.21.15.159:0/2353080594 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae400bb20 con 0x7f7aec071420 2024-01-15T14:34:40.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.349+0000 7f7aeaffd700 1 -- 172.21.15.159:0/2353080594 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400bed0 con 0x7f7aec071420 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.349+0000 7f7af2845700 1 -- 172.21.15.159:0/2353080594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec071420 msgr2=0x7f7aec071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.349+0000 7f7af2845700 1 --2- 172.21.15.159:0/2353080594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec071420 0x7f7aec071870 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009dc0 tx=0x7f7ae400b330 comp rx=0 tx=0).stop 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.350+0000 7f7af2845700 1 -- 172.21.15.159:0/2353080594 shutdown_connections 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.350+0000 7f7af2845700 1 --2- 172.21.15.159:0/2353080594 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec071420 0x7f7aec071870 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.350+0000 7f7af2845700 1 -- 172.21.15.159:0/2353080594 >> 172.21.15.159:0/2353080594 conn(0x7f7aec074170 msgr2=0x7f7aec06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.350+0000 7f7af2845700 1 -- 172.21.15.159:0/2353080594 shutdown_connections 2024-01-15T14:34:40.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.350+0000 7f7af2845700 1 -- 172.21.15.159:0/2353080594 wait complete. 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.351+0000 7f7af2845700 1 Processor -- start 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.351+0000 7f7af2845700 1 -- start start 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af2845700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 0x7f7aec1560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af2845700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec071420 0x7f7aec15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af2845700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7aec15a3e0 0x7f7aec158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af2845700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0ad870 con 0x7f7aec071420 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af2845700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0ae6d0 con 0x7f7aec15a3e0 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af2845700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec1e4de0 con 0x7f7aec06eba0 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 0x7f7aec1560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7af0de2700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7aec15a3e0 0x7f7aec158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 0x7f7aec1560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:46264/0 (socket says 172.21.15.159:46264) 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 -- 172.21.15.159:0/2342932407 learned_addr learned my addr 172.21.15.159:0/2342932407 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:40.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aeb7fe700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec071420 0x7f7aec15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 -- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7aec15a3e0 msgr2=0x7f7aec158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7aec15a3e0 0x7f7aec158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 -- 172.21.15.159:0/2342932407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec071420 msgr2=0x7f7aec15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec071420 0x7f7aec15a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 -- 172.21.15.159:0/2342932407 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4009a70 con 0x7f7aec06eba0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7aebfff700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 0x7f7aec1560e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00c550 tx=0x7f7adc00c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.352+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0054e0 con 0x7f7aec06eba0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.353+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc003680 con 0x7f7aec06eba0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.353+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc003a30 con 0x7f7aec06eba0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.353+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec0adac0 con 0x7f7aec06eba0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.353+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec0ae080 con 0x7f7aec06eba0 2024-01-15T14:34:40.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.353+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7adc01c440 con 0x7f7aec06eba0 2024-01-15T14:34:40.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.353+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f7adc0127b0 con 0x7f7aec06eba0 2024-01-15T14:34:40.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.354+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aec069020 con 0x7f7aec06eba0 2024-01-15T14:34:40.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.356+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7adc017070 con 0x7f7aec06eba0 2024-01-15T14:34:40.373 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.373+0000 7f95e115d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T14:34:40.440 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.439+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4079025f-8ebd-4dc4-a2b2-146beab483f2", "id": 1} v 0) v1 -- 0x7f7aec0b3c20 con 0x7f7aec06eba0 2024-01-15T14:34:40.445 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.445+0000 7f7ae8ff9700 1 -- 172.21.15.159:0/2342932407 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4079025f-8ebd-4dc4-a2b2-146beab483f2", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7aec071db0 con 0x7f7aec06eba0 2024-01-15T14:34:40.445 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-01-15T14:34:40.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 msgr2=0x7f7aec1560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 0x7f7aec1560e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00c550 tx=0x7f7adc00c860 comp rx=0 tx=0).stop 2024-01-15T14:34:40.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 shutdown_connections 2024-01-15T14:34:40.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7aec06eba0 0x7f7aec1560e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7aec15a3e0 0x7f7aec158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 --2- 172.21.15.159:0/2342932407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec071420 0x7f7aec15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.448 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 >> 172.21.15.159:0/2342932407 conn(0x7f7aec074170 msgr2=0x7f7aec156a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.448 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 shutdown_connections 2024-01-15T14:34:40.448 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.446+0000 7f7af2845700 1 -- 172.21.15.159:0/2342932407 wait complete. 2024-01-15T14:34:40.448 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.448+0000 7fec277c6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T14:34:40.459 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new c22b946b-abe1-42da-b00e-2c272bef68a5 2 2024-01-15T14:34:40.540 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.539+0000 7f95e115d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T14:34:40.543 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.542+0000 7fec277c6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T14:34:40.581 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.580+0000 7f072c3d5700 1 Processor -- start 2024-01-15T14:34:40.581 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.580+0000 7f072c3d5700 1 -- start start 2024-01-15T14:34:40.584 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.581+0000 7f072c3d5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f072406f680 0x7f07240afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.584 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.581+0000 7f072c3d5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072406fb30 con 0x7f072406e420 2024-01-15T14:34:40.584 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.581+0000 7f072c3d5700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072406d980 con 0x7f072415e110 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.581+0000 7f072c3d5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07240b0070 con 0x7f072406f680 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072a171700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f072415e110 0x7f072406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:40834/0 (socket says 172.21.15.159:40834) 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072a171700 1 -- 172.21.15.159:0/614080339 learned_addr learned my addr 172.21.15.159:0/614080339 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f0729970700 1 --2- 172.21.15.159:0/614080339 >> v2:172.21.15.77:3300/0 conn(0x7f072406f680 0x7f07240afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072916f700 1 -- 172.21.15.159:0/614080339 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373270118 0 0) 0x7f072406d980 con 0x7f072415e110 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072916f700 1 -- 172.21.15.159:0/614080339 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07100035b0 con 0x7f072415e110 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f0729970700 1 -- 172.21.15.159:0/614080339 >> v1:172.21.15.159:6789/0 conn(0x7f072415e110 legacy=0x7f072406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f0729970700 1 -- 172.21.15.159:0/614080339 >> v1:172.21.15.77:6789/0 conn(0x7f072406e420 legacy=0x7f0724073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072916f700 1 -- 172.21.15.159:0/614080339 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 332970957 0 0) 0x7f07100035b0 con 0x7f072415e110 2024-01-15T14:34:40.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f0729970700 1 -- 172.21.15.159:0/614080339 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240b0260 con 0x7f072406f680 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f0729970700 1 --2- 172.21.15.159:0/614080339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240afb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f07140097c0 tx=0x7f0714009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06428a9fdc62563 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072916f700 1 -- 172.21.15.159:0/614080339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720001670 con 0x7f072406f680 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072916f700 1 -- 172.21.15.159:0/614080339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0714004030 con 0x7f072406f680 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.582+0000 7f072916f700 1 -- 172.21.15.159:0/614080339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07140043c0 con 0x7f072406f680 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 -- 172.21.15.159:0/614080339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 msgr2=0x7f07240afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 --2- 172.21.15.159:0/614080339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240afb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f07140097c0 tx=0x7f0714009ad0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 -- 172.21.15.159:0/614080339 shutdown_connections 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 --2- 172.21.15.159:0/614080339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240afb30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 -- 172.21.15.159:0/614080339 >> 172.21.15.159:0/614080339 conn(0x7f07240b3250 msgr2=0x7f07240d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.586 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 -- 172.21.15.159:0/614080339 shutdown_connections 2024-01-15T14:34:40.587 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.583+0000 7f072c3d5700 1 -- 172.21.15.159:0/614080339 wait complete. 2024-01-15T14:34:40.589 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.584+0000 7f072c3d5700 1 Processor -- start 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.584+0000 7f072c3d5700 1 -- start start 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072c3d5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 0x7f07241f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072c3d5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072c3d5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f072415e110 0x7f07240c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072c3d5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07240c60a0 con 0x7f072406f680 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072c3d5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07240c62d0 con 0x7f072406e420 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072c3d5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07240c65b0 con 0x7f072415e110 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 0x7f07241f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.590 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a972700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f072415e110 0x7f07240c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 0x7f07241f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37058/0 (socket says 172.21.15.159:37058) 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 -- 172.21.15.159:0/1247389597 learned_addr learned my addr 172.21.15.159:0/1247389597 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f0729970700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 -- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f072415e110 msgr2=0x7f07240c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f072415e110 0x7f07240c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 -- 172.21.15.159:0/1247389597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 msgr2=0x7f07240c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.585+0000 7f072a171700 1 -- 172.21.15.159:0/1247389597 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140094a0 con 0x7f072406e420 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f072a171700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 0x7f07241f8030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f07200011e0 tx=0x7f072000c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.591 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720004c80 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0720004e00 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720007310 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07240c6890 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07240c6e50 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f072001d3f0 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.586+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f07200137e0 con 0x7f072406e420 2024-01-15T14:34:40.592 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.587+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07241615a0 con 0x7f072406e420 2024-01-15T14:34:40.593 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.591+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0720013470 con 0x7f072406e420 2024-01-15T14:34:40.619 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.618+0000 7fec277c6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T14:34:40.676 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.676+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c22b946b-abe1-42da-b00e-2c272bef68a5", "id": 2} v 0) v1 -- 0x7f0724069020 con 0x7f072406e420 2024-01-15T14:34:40.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.681+0000 7f071affd700 1 -- 172.21.15.159:0/1247389597 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c22b946b-abe1-42da-b00e-2c272bef68a5", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0720013a40 con 0x7f072406e420 2024-01-15T14:34:40.682 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-01-15T14:34:40.683 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 msgr2=0x7f07241f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.683 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 0x7f07241f8030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f07200011e0 tx=0x7f072000c9e0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.683 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 shutdown_connections 2024-01-15T14:34:40.683 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f072415e110 0x7f07240c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.684 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f072406e420 0x7f07241f8030 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.684 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 --2- 172.21.15.159:0/1247389597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f072406f680 0x7f07240c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.684 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 >> 172.21.15.159:0/1247389597 conn(0x7f07240b3250 msgr2=0x7f072415e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.684 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 shutdown_connections 2024-01-15T14:34:40.684 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.682+0000 7f072c3d5700 1 -- 172.21.15.159:0/1247389597 wait complete. 2024-01-15T14:34:40.694 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 2874d645-bf49-4305-bdcb-8368258f238b 3 2024-01-15T14:34:40.702 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.702+0000 7f95e115d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T14:34:40.818 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.818+0000 7f57db620700 1 Processor -- start 2024-01-15T14:34:40.818 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.818+0000 7f57db620700 1 -- start start 2024-01-15T14:34:40.821 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57db620700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f57d40ceb20 0x7f57d4166af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.821 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57db620700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d40d0790 con 0x7f57d40d01b0 2024-01-15T14:34:40.821 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57db620700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d406ee70 con 0x7f57d415bee0 2024-01-15T14:34:40.821 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57db620700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d406f150 con 0x7f57d40ceb20 2024-01-15T14:34:40.821 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d9bbd700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f57d40ceb20 0x7f57d4166af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d93bc700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f57d415bee0 0x7f57d40cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:40840/0 (socket says 172.21.15.159:40840) 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d93bc700 1 -- 172.21.15.159:0/4218393064 learned_addr learned my addr 172.21.15.159:0/4218393064 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d9bbd700 1 -- 172.21.15.159:0/4218393064 >> v1:172.21.15.159:6789/0 conn(0x7f57d415bee0 legacy=0x7f57d40cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d9bbd700 1 -- 172.21.15.159:0/4218393064 >> v1:172.21.15.77:6789/0 conn(0x7f57d40d01b0 legacy=0x7f57d40ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d9bbd700 1 -- 172.21.15.159:0/4218393064 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d406f430 con 0x7f57d40ceb20 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57d9bbd700 1 --2- 172.21.15.159:0/4218393064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d40ceb20 0x7f57d4166af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009d90 tx=0x7f57cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69220fa8d1dd744a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.819+0000 7f57cbfff700 1 -- 172.21.15.159:0/4218393064 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00e070 con 0x7f57d40ceb20 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57cbfff700 1 -- 172.21.15.159:0/4218393064 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc00bbd0 con 0x7f57d40ceb20 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57cbfff700 1 -- 172.21.15.159:0/4218393064 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc012430 con 0x7f57d40ceb20 2024-01-15T14:34:40.822 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 -- 172.21.15.159:0/4218393064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d40ceb20 msgr2=0x7f57d4166af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 --2- 172.21.15.159:0/4218393064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d40ceb20 0x7f57d4166af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009d90 tx=0x7f57cc00b330 comp rx=0 tx=0).stop 2024-01-15T14:34:40.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 -- 172.21.15.159:0/4218393064 shutdown_connections 2024-01-15T14:34:40.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 --2- 172.21.15.159:0/4218393064 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d40ceb20 0x7f57d4166af0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 -- 172.21.15.159:0/4218393064 >> 172.21.15.159:0/4218393064 conn(0x7f57d40b3780 msgr2=0x7f57d40b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 -- 172.21.15.159:0/4218393064 shutdown_connections 2024-01-15T14:34:40.823 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.820+0000 7f57db620700 1 -- 172.21.15.159:0/4218393064 wait complete. 2024-01-15T14:34:40.824 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.822+0000 7f57db620700 1 Processor -- start 2024-01-15T14:34:40.824 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.822+0000 7f57db620700 1 -- start start 2024-01-15T14:34:40.824 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.822+0000 7f95e115d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57db620700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d40ceb20 0x7f57d415f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57db620700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57d40d01b0 0x7f57d41f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57db620700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 0x7f57d40caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57db620700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d415d410 con 0x7f57d40ceb20 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57db620700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d415c5b0 con 0x7f57d415bee0 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57db620700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d415c890 con 0x7f57d40d01b0 2024-01-15T14:34:40.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57d9bbd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 0x7f57d40caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57d9bbd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 0x7f57d40caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37080/0 (socket says 172.21.15.159:37080) 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57d8bbb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57d40d01b0 0x7f57d41f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.823+0000 7f57d9bbd700 1 -- 172.21.15.159:0/68850416 learned_addr learned my addr 172.21.15.159:0/68850416 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d93bc700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d40ceb20 0x7f57d415f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d9bbd700 1 -- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57d40d01b0 msgr2=0x7f57d41f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d9bbd700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57d40d01b0 0x7f57d41f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d9bbd700 1 -- 172.21.15.159:0/68850416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d40ceb20 msgr2=0x7f57d415f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d9bbd700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d40ceb20 0x7f57d415f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d9bbd700 1 -- 172.21.15.159:0/68850416 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc009a70 con 0x7f57d415bee0 2024-01-15T14:34:40.829 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d93bc700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d40ceb20 0x7f57d415f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57d9bbd700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 0x7f57d40caac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f57cc014040 tx=0x7f57cc004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc012dd0 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc020070 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc004280 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d415cb70 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.824+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d415d130 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.825+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f57cc0046e0 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.825+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f57cc022070 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.825+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4069020 con 0x7f57d415bee0 2024-01-15T14:34:40.830 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.827+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f57cc02c060 con 0x7f57d415bee0 2024-01-15T14:34:40.865 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.864+0000 7fec277c6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T14:34:40.912 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.911+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2874d645-bf49-4305-bdcb-8368258f238b", "id": 3} v 0) v1 -- 0x7f57d415ac40 con 0x7f57d415bee0 2024-01-15T14:34:40.917 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.917+0000 7f57c9ffb700 1 -- 172.21.15.159:0/68850416 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2874d645-bf49-4305-bdcb-8368258f238b", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f57cc027480 con 0x7f57d415bee0 2024-01-15T14:34:40.917 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 msgr2=0x7f57d40caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 0x7f57d40caac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f57cc014040 tx=0x7f57cc004ad0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 shutdown_connections 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f57d415bee0 0x7f57d40caac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d40ceb20 0x7f57d415f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 --2- 172.21.15.159:0/68850416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f57d40d01b0 0x7f57d41f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 >> 172.21.15.159:0/68850416 conn(0x7f57d40b3780 msgr2=0x7f57d4061c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 shutdown_connections 2024-01-15T14:34:40.919 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:40.918+0000 7f57db620700 1 -- 172.21.15.159:0/68850416 wait complete. 2024-01-15T14:34:40.921 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:40.921+0000 7f95e115d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T14:34:40.930 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 722287a8-22c1-4d04-92a0-2e538826f8f3 4 2024-01-15T14:34:40.935 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:40.935+0000 7fec277c6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T14:34:41.001 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:41.001+0000 7f95e115d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T14:34:41.093 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.093+0000 7fbc94598700 1 Processor -- start 2024-01-15T14:34:41.094 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.093+0000 7fbc94598700 1 -- start start 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc94598700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbc8c073f90 0x7fbc8c0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc94598700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0b42a0 con 0x7fbc8c0b5220 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc94598700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0af900 con 0x7fbc8c15e280 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc94598700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c074d30 con 0x7fbc8c073f90 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc92b35700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbc8c073f90 0x7fbc8c0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc92334700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbc8c15e280 0x7fbc8c0b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:40852/0 (socket says 172.21.15.159:40852) 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc92334700 1 -- 172.21.15.159:0/4282855510 learned_addr learned my addr 172.21.15.159:0/4282855510 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc92b35700 1 -- 172.21.15.159:0/4282855510 >> v1:172.21.15.159:6789/0 conn(0x7fbc8c15e280 legacy=0x7fbc8c0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc92b35700 1 -- 172.21.15.159:0/4282855510 >> v1:172.21.15.77:6789/0 conn(0x7fbc8c0b5220 legacy=0x7fbc8c06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.098 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc91332700 1 -- 172.21.15.159:0/4282855510 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399378932 0 0) 0x7fbc8c0af900 con 0x7fbc8c15e280 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.094+0000 7fbc92b35700 1 -- 172.21.15.159:0/4282855510 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c074490 con 0x7fbc8c073f90 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc92b35700 1 --2- 172.21.15.159:0/4282855510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c0b3d60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009d90 tx=0x7fbc8400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aa1d5b8a326e674 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc91332700 1 -- 172.21.15.159:0/4282855510 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8400e070 con 0x7fbc8c073f90 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc91332700 1 -- 172.21.15.159:0/4282855510 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8400bbd0 con 0x7fbc8c073f90 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc91332700 1 -- 172.21.15.159:0/4282855510 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84012430 con 0x7fbc8c073f90 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 -- 172.21.15.159:0/4282855510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 msgr2=0x7fbc8c0b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 --2- 172.21.15.159:0/4282855510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c0b3d60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009d90 tx=0x7fbc8400b330 comp rx=0 tx=0).stop 2024-01-15T14:34:41.099 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 -- 172.21.15.159:0/4282855510 shutdown_connections 2024-01-15T14:34:41.100 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 --2- 172.21.15.159:0/4282855510 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c0b3d60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.100 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 -- 172.21.15.159:0/4282855510 >> 172.21.15.159:0/4282855510 conn(0x7fbc8c072b20 msgr2=0x7fbc8c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.100 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 -- 172.21.15.159:0/4282855510 shutdown_connections 2024-01-15T14:34:41.100 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.095+0000 7fbc94598700 1 -- 172.21.15.159:0/4282855510 wait complete. 2024-01-15T14:34:41.100 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.097+0000 7fbc94598700 1 Processor -- start 2024-01-15T14:34:41.100 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.097+0000 7fbc94598700 1 -- start start 2024-01-15T14:34:41.102 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc94598700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c14c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.102 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc94598700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc8c0b5220 0x7fbc8c152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.102 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc94598700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc8c153830 0x7fbc8c0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.102 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc94598700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c153d30 con 0x7fbc8c0b5220 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc94598700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c154010 con 0x7fbc8c073f90 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc94598700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c150930 con 0x7fbc8c153830 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc92334700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c14c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc92334700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c14c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37090/0 (socket says 172.21.15.159:37090) 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.098+0000 7fbc92334700 1 -- 172.21.15.159:0/2703866809 learned_addr learned my addr 172.21.15.159:0/2703866809 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc91b33700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc8c0b5220 0x7fbc8c152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc92334700 1 -- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc8c153830 msgr2=0x7fbc8c0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc92334700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc8c153830 0x7fbc8c0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.103 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc92334700 1 -- 172.21.15.159:0/2703866809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc8c0b5220 msgr2=0x7fbc8c152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc92334700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc8c0b5220 0x7fbc8c152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc92334700 1 -- 172.21.15.159:0/2703866809 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc84009a70 con 0x7fbc8c073f90 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc91b33700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc8c0b5220 0x7fbc8c152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc92334700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c14c9f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800c520 tx=0x7fbc8800c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88004c80 con 0x7fbc8c073f90 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc88004e20 con 0x7fbc8c073f90 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88007310 con 0x7fbc8c073f90 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c150c10 con 0x7fbc8c073f90 2024-01-15T14:34:41.104 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.099+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c14cf30 con 0x7fbc8c073f90 2024-01-15T14:34:41.105 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.100+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbc880133c0 con 0x7fbc8c073f90 2024-01-15T14:34:41.105 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.101+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fbc88013b80 con 0x7fbc8c073f90 2024-01-15T14:34:41.105 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.101+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc740052f0 con 0x7fbc8c073f90 2024-01-15T14:34:41.105 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.104+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fbc88013d40 con 0x7fbc8c073f90 2024-01-15T14:34:41.219 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.219+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "722287a8-22c1-4d04-92a0-2e538826f8f3", "id": 4} v 0) v1 -- 0x7fbc74005640 con 0x7fbc8c073f90 2024-01-15T14:34:41.223 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.222+0000 7fbc82ffd700 1 -- 172.21.15.159:0/2703866809 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "722287a8-22c1-4d04-92a0-2e538826f8f3", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbc88013670 con 0x7fbc8c073f90 2024-01-15T14:34:41.223 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-01-15T14:34:41.224 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 msgr2=0x7fbc8c14c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.224 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c14c9f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800c520 tx=0x7fbc8800c8e0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 shutdown_connections 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbc8c153830 0x7fbc8c0b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbc8c073f90 0x7fbc8c14c9f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 --2- 172.21.15.159:0/2703866809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc8c0b5220 0x7fbc8c152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 >> 172.21.15.159:0/2703866809 conn(0x7fbc8c072b20 msgr2=0x7fbc8c0ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 shutdown_connections 2024-01-15T14:34:41.225 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.223+0000 7fbc94598700 1 -- 172.21.15.159:0/2703866809 wait complete. 2024-01-15T14:34:41.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 47ab64b1-a18f-42dd-875a-3df51f073d8c 5 2024-01-15T14:34:41.240 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:41.239+0000 7f95e115d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T14:34:41.318 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:41.317+0000 7f95e115d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T14:34:41.322 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:41.321+0000 7fec277c6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T14:34:41.364 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.364+0000 7fb4925e1700 1 Processor -- start 2024-01-15T14:34:41.365 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.364+0000 7fb4925e1700 1 -- start start 2024-01-15T14:34:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb4925e1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb4925e1700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb48c0b0170 0x7fb48c074410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb4925e1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c145b70 con 0x7fb48c073f90 2024-01-15T14:34:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb4925e1700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48c0b05e0 con 0x7fb48c0b0170 2024-01-15T14:34:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb4925e1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c074950 con 0x7fb48c145790 2024-01-15T14:34:41.368 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:46310/0 (socket says 172.21.15.159:46310) 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 -- 172.21.15.159:0/1756618135 learned_addr learned my addr 172.21.15.159:0/1756618135 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb490b7e700 1 --2- 172.21.15.159:0/1756618135 >> v2:172.21.15.77:3300/0 conn(0x7fb48c0b0170 0x7fb48c074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 -- 172.21.15.159:0/1756618135 >> v2:172.21.15.77:3300/0 conn(0x7fb48c0b0170 msgr2=0x7fb48c074410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 --2- 172.21.15.159:0/1756618135 >> v2:172.21.15.77:3300/0 conn(0x7fb48c0b0170 0x7fb48c074410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 -- 172.21.15.159:0/1756618135 >> v1:172.21.15.77:6789/0 conn(0x7fb48c073f90 legacy=0x7fb48c0afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 -- 172.21.15.159:0/1756618135 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0b3bf0 con 0x7fb48c145790 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48bfff700 1 --2- 172.21.15.159:0/1756618135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c0b4c00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0098f0 tx=0x7fb47c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee2adf32c8aa6b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48affd700 1 -- 172.21.15.159:0/1756618135 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c00b8a0 con 0x7fb48c145790 2024-01-15T14:34:41.369 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48affd700 1 -- 172.21.15.159:0/1756618135 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47c00ba00 con 0x7fb48c145790 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.365+0000 7fb48affd700 1 -- 172.21.15.159:0/1756618135 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c00bd90 con 0x7fb48c145790 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 -- 172.21.15.159:0/1756618135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 msgr2=0x7fb48c0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 --2- 172.21.15.159:0/1756618135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c0b4c00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0098f0 tx=0x7fb47c009c00 comp rx=0 tx=0).stop 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 -- 172.21.15.159:0/1756618135 shutdown_connections 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 --2- 172.21.15.159:0/1756618135 >> v2:172.21.15.77:3300/0 conn(0x7fb48c0b0170 0x7fb48c074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 --2- 172.21.15.159:0/1756618135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c0b4c00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 -- 172.21.15.159:0/1756618135 >> 172.21.15.159:0/1756618135 conn(0x7fb48c0b3250 msgr2=0x7fb48c0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 -- 172.21.15.159:0/1756618135 shutdown_connections 2024-01-15T14:34:41.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.366+0000 7fb4925e1700 1 -- 172.21.15.159:0/1756618135 wait complete. 2024-01-15T14:34:41.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.367+0000 7fb4925e1700 1 Processor -- start 2024-01-15T14:34:41.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.367+0000 7fb4925e1700 1 -- start start 2024-01-15T14:34:41.376 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb4925e1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 0x7fb48c140790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.376 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb4925e1700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb48c0b0170 0x7fb48c15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.376 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb4925e1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.376 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb4925e1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c1f6da0 con 0x7fb48c073f90 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb4925e1700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c15d110 con 0x7fb48c0b0170 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb4925e1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c1f3c50 con 0x7fb48c145790 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb48b7fe700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb48c0b0170 0x7fb48c15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb490b7e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb48bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 0x7fb48c140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb490b7e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c06d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:46312/0 (socket says 172.21.15.159:46312) 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb48bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 0x7fb48c140790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.159:33762/0 (socket says 172.21.15.159:33762) 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.368+0000 7fb490b7e700 1 -- 172.21.15.159:0/2918650900 learned_addr learned my addr 172.21.15.159:0/2918650900 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb490b7e700 1 -- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb48c0b0170 msgr2=0x7fb48c15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.377 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb490b7e700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb48c0b0170 0x7fb48c15cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb490b7e700 1 -- 172.21.15.159:0/2918650900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 msgr2=0x7fb48c140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb490b7e700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 0x7fb48c140790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb490b7e700 1 -- 172.21.15.159:0/2918650900 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c009550 con 0x7fb48c145790 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb48bfff700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 0x7fb48c140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb490b7e700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c06d980 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb484007e40 tx=0x7fb48400e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400ed30 con 0x7fb48c145790 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb484014420 con 0x7fb48c145790 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c1f3f30 con 0x7fb48c145790 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4840147d0 con 0x7fb48c145790 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c1f44f0 con 0x7fb48c145790 2024-01-15T14:34:41.378 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb484014c30 con 0x7fb48c145790 2024-01-15T14:34:41.379 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.369+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fb48401ec80 con 0x7fb48c145790 2024-01-15T14:34:41.379 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.370+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48c1f6f10 con 0x7fb48c145790 2024-01-15T14:34:41.379 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.372+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb48401e510 con 0x7fb48c145790 2024-01-15T14:34:41.457 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.457+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "47ab64b1-a18f-42dd-875a-3df51f073d8c", "id": 5} v 0) v1 -- 0x7fb48c069020 con 0x7fb48c145790 2024-01-15T14:34:41.465 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.465+0000 7fb488ff9700 1 -- 172.21.15.159:0/2918650900 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47ab64b1-a18f-42dd-875a-3df51f073d8c", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb48401e6b0 con 0x7fb48c145790 2024-01-15T14:34:41.466 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-01-15T14:34:41.467 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 msgr2=0x7fb48c06d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.467 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c06d980 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb484007e40 tx=0x7fb48400e3c0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.467 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 shutdown_connections 2024-01-15T14:34:41.467 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb48c145790 0x7fb48c06d980 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.467 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48c073f90 0x7fb48c140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.467 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 --2- 172.21.15.159:0/2918650900 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb48c0b0170 0x7fb48c15cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.468 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 >> 172.21.15.159:0/2918650900 conn(0x7fb48c0b3250 msgr2=0x7fb48c0b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.468 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 shutdown_connections 2024-01-15T14:34:41.468 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.466+0000 7fb4925e1700 1 -- 172.21.15.159:0/2918650900 wait complete. 2024-01-15T14:34:41.478 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 3c2335e5-12aa-4141-8e1d-628092b4dfb4 6 2024-01-15T14:34:41.543 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:41.542+0000 7fec277c6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T14:34:41.604 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.603+0000 7f33f404f700 1 Processor -- start 2024-01-15T14:34:41.604 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.603+0000 7f33f404f700 1 -- start start 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f404f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 0x7f33ec0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f404f700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 0x7f33ec0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f404f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ec0d0790 con 0x7f33ec0ce950 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f404f700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec0cedc0 con 0x7f33ec0cf020 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f404f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec06ee70 con 0x7f33ec15bee0 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f15ea700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 0x7f33ec0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f1deb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 0x7f33ec0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f15ea700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 0x7f33ec0ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37100/0 (socket says 172.21.15.159:37100) 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f15ea700 1 -- 172.21.15.159:0/3577931300 learned_addr learned my addr 172.21.15.159:0/3577931300 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.604+0000 7f33f1deb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 0x7f33ec0d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:46320/0 (socket says 172.21.15.159:46320) 2024-01-15T14:34:41.607 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f1deb700 1 -- 172.21.15.159:0/3577931300 >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 msgr2=0x7f33ec0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f1deb700 1 --2- 172.21.15.159:0/3577931300 >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 0x7f33ec0ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f1deb700 1 -- 172.21.15.159:0/3577931300 >> v1:172.21.15.77:6789/0 conn(0x7f33ec0ce950 legacy=0x7f33ec0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f1deb700 1 -- 172.21.15.159:0/3577931300 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec06f150 con 0x7f33ec15bee0 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f15ea700 1 --2- 172.21.15.159:0/3577931300 >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 0x7f33ec0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f1deb700 1 --2- 172.21.15.159:0/3577931300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 0x7f33ec0d01b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009840 tx=0x7f33dc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48757e38118e751c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f0de9700 1 -- 172.21.15.159:0/3577931300 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc011830 con 0x7f33ec15bee0 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f0de9700 1 -- 172.21.15.159:0/3577931300 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc011990 con 0x7f33ec15bee0 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.605+0000 7f33f0de9700 1 -- 172.21.15.159:0/3577931300 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc011d20 con 0x7f33ec15bee0 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 -- 172.21.15.159:0/3577931300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 msgr2=0x7f33ec0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 --2- 172.21.15.159:0/3577931300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 0x7f33ec0d01b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009840 tx=0x7f33dc009b50 comp rx=0 tx=0).stop 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 -- 172.21.15.159:0/3577931300 shutdown_connections 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 --2- 172.21.15.159:0/3577931300 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec15bee0 0x7f33ec0d01b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.608 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 --2- 172.21.15.159:0/3577931300 >> v2:172.21.15.159:3300/0 conn(0x7f33ec0cf020 0x7f33ec0ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 -- 172.21.15.159:0/3577931300 >> 172.21.15.159:0/3577931300 conn(0x7f33ec0b3780 msgr2=0x7f33ec0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 -- 172.21.15.159:0/3577931300 shutdown_connections 2024-01-15T14:34:41.609 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.606+0000 7f33f404f700 1 -- 172.21.15.159:0/3577931300 wait complete. 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.607+0000 7f33f404f700 1 Processor -- start 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.607+0000 7f33f404f700 1 -- start start 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f404f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec0ce950 0x7f33ec150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f404f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 0x7f33ec0cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f404f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 0x7f33ec15e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f404f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0ca440 con 0x7f33ec0ce950 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f404f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0c9830 con 0x7f33ec15bee0 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f404f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0c9b10 con 0x7f33ec0cf020 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f15ea700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 0x7f33ec0cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 0x7f33ec15e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.612 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f15ea700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 0x7f33ec0cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.159:46322/0 (socket says 172.21.15.159:46322) 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 0x7f33ec15e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37112/0 (socket says 172.21.15.159:37112) 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f15ea700 1 -- 172.21.15.159:0/417819048 learned_addr learned my addr 172.21.15.159:0/417819048 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f1deb700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec0ce950 0x7f33ec150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 -- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 msgr2=0x7f33ec0cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 0x7f33ec0cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 -- 172.21.15.159:0/417819048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec0ce950 msgr2=0x7f33ec150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec0ce950 0x7f33ec150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.608+0000 7f33f25ec700 1 -- 172.21.15.159:0/417819048 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e8009550 con 0x7f33ec15bee0 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33f15ea700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 0x7f33ec0cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33f25ec700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 0x7f33ec15e440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f33e400cae0 tx=0x7f33e400cdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e4003c00 con 0x7f33ec15bee0 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e4003d60 con 0x7f33ec15bee0 2024-01-15T14:34:41.613 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e400aa20 con 0x7f33ec15bee0 2024-01-15T14:34:41.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc0094a0 con 0x7f33ec15bee0 2024-01-15T14:34:41.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec0ca0d0 con 0x7f33ec15bee0 2024-01-15T14:34:41.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f33e40050d0 con 0x7f33ec15bee0 2024-01-15T14:34:41.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.609+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f33e4012470 con 0x7f33ec15bee0 2024-01-15T14:34:41.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.610+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec0ca290 con 0x7f33ec15bee0 2024-01-15T14:34:41.614 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.612+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f33e4005380 con 0x7f33ec15bee0 2024-01-15T14:34:41.698 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.697+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3c2335e5-12aa-4141-8e1d-628092b4dfb4", "id": 6} v 0) v1 -- 0x7f33ec069020 con 0x7f33ec15bee0 2024-01-15T14:34:41.706 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.706+0000 7f33e27fc700 1 -- 172.21.15.159:0/417819048 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c2335e5-12aa-4141-8e1d-628092b4dfb4", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f33e40055f0 con 0x7f33ec15bee0 2024-01-15T14:34:41.706 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 msgr2=0x7f33ec15e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 0x7f33ec15e440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f33e400cae0 tx=0x7f33e400cdf0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 shutdown_connections 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f33ec15bee0 0x7f33ec15e440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec0ce950 0x7f33ec150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 --2- 172.21.15.159:0/417819048 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f33ec0cf020 0x7f33ec0cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 >> 172.21.15.159:0/417819048 conn(0x7f33ec0b3780 msgr2=0x7f33ec0b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 shutdown_connections 2024-01-15T14:34:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.707+0000 7f33f404f700 1 -- 172.21.15.159:0/417819048 wait complete. 2024-01-15T14:34:41.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 5154d400-3a7b-45d9-8e34-2d3057cabd8f 7 2024-01-15T14:34:41.725 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:41.725+0000 7f95e115d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T14:34:41.776 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:41.775+0000 7fec277c6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-15T14:34:41.843 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.843+0000 7f720bd81700 1 Processor -- start 2024-01-15T14:34:41.843 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.843+0000 7f720bd81700 1 -- start start 2024-01-15T14:34:41.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720bd81700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f720406f680 0x7f72040afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720bd81700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720406e420 0x7f720406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720bd81700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720406fb30 con 0x7f720415e110 2024-01-15T14:34:41.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720bd81700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72040b0070 con 0x7f720406f680 2024-01-15T14:34:41.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720bd81700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720406d980 con 0x7f720406e420 2024-01-15T14:34:41.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720a31e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720406e420 0x7f720406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f7209b1d700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f720415e110 0x7f720406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:40862/0 (socket says 172.21.15.159:40862) 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f7209b1d700 1 -- 172.21.15.159:0/639479708 learned_addr learned my addr 172.21.15.159:0/639479708 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720931c700 1 --2- 172.21.15.159:0/639479708 >> v2:172.21.15.77:3300/0 conn(0x7f720406f680 0x7f72040afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720a31e700 1 -- 172.21.15.159:0/639479708 >> v2:172.21.15.77:3300/0 conn(0x7f720406f680 msgr2=0x7f72040afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f7208b1b700 1 -- 172.21.15.159:0/639479708 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364308111 0 0) 0x7f720406fb30 con 0x7f720415e110 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720a31e700 1 --2- 172.21.15.159:0/639479708 >> v2:172.21.15.77:3300/0 conn(0x7f720406f680 0x7f72040afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720a31e700 1 -- 172.21.15.159:0/639479708 >> v1:172.21.15.159:6789/0 conn(0x7f720415e110 legacy=0x7f720406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720a31e700 1 -- 172.21.15.159:0/639479708 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040b0260 con 0x7f720406e420 2024-01-15T14:34:41.848 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720931c700 1 --2- 172.21.15.159:0/639479708 >> v2:172.21.15.77:3300/0 conn(0x7f720406f680 0x7f72040afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f720a31e700 1 --2- 172.21.15.159:0/639479708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720406e420 0x7f720406e8b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f71fc009d90 tx=0x7f71fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999df7de3d82eaef server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f7208b1b700 1 -- 172.21.15.159:0/639479708 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc00e070 con 0x7f720406e420 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f7208b1b700 1 -- 172.21.15.159:0/639479708 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71fc00bbd0 con 0x7f720406e420 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.844+0000 7f7208b1b700 1 -- 172.21.15.159:0/639479708 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc012430 con 0x7f720406e420 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 -- 172.21.15.159:0/639479708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720406e420 msgr2=0x7f720406e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 --2- 172.21.15.159:0/639479708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720406e420 0x7f720406e8b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f71fc009d90 tx=0x7f71fc00b330 comp rx=0 tx=0).stop 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 -- 172.21.15.159:0/639479708 shutdown_connections 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 --2- 172.21.15.159:0/639479708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720406e420 0x7f720406e8b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 --2- 172.21.15.159:0/639479708 >> v2:172.21.15.77:3300/0 conn(0x7f720406f680 0x7f72040afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 -- 172.21.15.159:0/639479708 >> 172.21.15.159:0/639479708 conn(0x7f72040b3250 msgr2=0x7f72040d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 -- 172.21.15.159:0/639479708 shutdown_connections 2024-01-15T14:34:41.850 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.845+0000 7f720bd81700 1 -- 172.21.15.159:0/639479708 wait complete. 2024-01-15T14:34:41.850 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.846+0000 7f720bd81700 1 Processor -- start 2024-01-15T14:34:41.850 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.846+0000 7f720bd81700 1 -- start start 2024-01-15T14:34:41.853 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720bd81700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f720406e420 0x7f720414c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.853 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720bd81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f720406f680 0x7f72040cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.853 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720bd81700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720415e110 0x7f72040cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:41.853 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720bd81700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72040c86b0 con 0x7f720406f680 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720bd81700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72040c83d0 con 0x7f720406e420 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720bd81700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720414cda0 con 0x7f720415e110 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f7209b1d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f720406e420 0x7f720414c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720a31e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720415e110 0x7f72040cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f720931c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f720406f680 0x7f72040cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f7209b1d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f720406e420 0x7f720414c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.159:37118/0 (socket says 172.21.15.159:37118) 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.847+0000 7f7209b1d700 1 -- 172.21.15.159:0/1049834373 learned_addr learned my addr 172.21.15.159:0/1049834373 (peer_addr_for_me v2:172.21.15.159:0/0) 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720a31e700 1 -- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f720406e420 msgr2=0x7f720414c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720a31e700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f720406e420 0x7f720414c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.854 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720a31e700 1 -- 172.21.15.159:0/1049834373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f720406f680 msgr2=0x7f72040cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720a31e700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f720406f680 0x7f72040cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720a31e700 1 -- 172.21.15.159:0/1049834373 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc009a70 con 0x7f720415e110 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720931c700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f720406f680 0x7f72040cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720a31e700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720415e110 0x7f72040cb760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71fc016000 tx=0x7f71fc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc012430 con 0x7f720415e110 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71fc020070 con 0x7f720415e110 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720414d080 con 0x7f720415e110 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc004280 con 0x7f720415e110 2024-01-15T14:34:41.855 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72040cd290 con 0x7f720415e110 2024-01-15T14:34:41.856 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f71fc0046e0 con 0x7f720415e110 2024-01-15T14:34:41.856 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.848+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f71fc022070 con 0x7f720415e110 2024-01-15T14:34:41.856 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.849+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720414cc00 con 0x7f720415e110 2024-01-15T14:34:41.856 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.851+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f71fc046ce0 con 0x7f720415e110 2024-01-15T14:34:41.856 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:41.853+0000 7fec277c6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T14:34:41.935 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.935+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5154d400-3a7b-45d9-8e34-2d3057cabd8f", "id": 7} v 0) v1 -- 0x7f720414cee0 con 0x7f720415e110 2024-01-15T14:34:41.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.940+0000 7f71fa7fc700 1 -- 172.21.15.159:0/1049834373 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5154d400-3a7b-45d9-8e34-2d3057cabd8f", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f720414cee0 con 0x7f720415e110 2024-01-15T14:34:41.941 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-01-15T14:34:41.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720415e110 msgr2=0x7f72040cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:41.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720415e110 0x7f72040cb760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71fc016000 tx=0x7f71fc004ad0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 shutdown_connections 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f720415e110 0x7f72040cb760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f720406e420 0x7f720414c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 --2- 172.21.15.159:0/1049834373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f720406f680 0x7f72040cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 >> 172.21.15.159:0/1049834373 conn(0x7f72040b3250 msgr2=0x7f72040b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 shutdown_connections 2024-01-15T14:34:41.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-01-15T14:34:41.941+0000 7f720bd81700 1 -- 172.21.15.159:0/1049834373 wait complete. 2024-01-15T14:34:41.954 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-15T14:34:41.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-15T14:34:41.956 INFO:tasks.ceph.osd.0:Started 2024-01-15T14:34:41.957 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-15T14:34:41.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-15T14:34:41.958 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:41.957+0000 7f95e115d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T14:34:41.960 INFO:tasks.ceph.osd.1:Started 2024-01-15T14:34:41.960 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-15T14:34:41.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-15T14:34:41.963 INFO:tasks.ceph.osd.2:Started 2024-01-15T14:34:41.963 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-15T14:34:41.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-15T14:34:41.966 INFO:tasks.ceph.osd.3:Started 2024-01-15T14:34:41.966 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-15T14:34:41.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-15T14:34:41.968 INFO:tasks.ceph.osd.4:Started 2024-01-15T14:34:41.969 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-15T14:34:41.969 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-15T14:34:41.971 INFO:tasks.ceph.osd.5:Started 2024-01-15T14:34:41.971 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-15T14:34:41.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-15T14:34:41.974 INFO:tasks.ceph.osd.6:Started 2024-01-15T14:34:41.974 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-15T14:34:41.975 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-15T14:34:41.979 INFO:tasks.ceph.osd.7:Started 2024-01-15T14:34:41.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:34:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.143+0000 7f326429c700 1 Processor -- start 2024-01-15T14:34:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.143+0000 7f326429c700 1 -- start start 2024-01-15T14:34:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f326429c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f326429c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f325c0d2160 0x7f325c0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f326429c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c0d0770 con 0x7f325c1603c0 2024-01-15T14:34:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f326429c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c15c2b0 con 0x7f325c0d2160 2024-01-15T14:34:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f326429c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0cd100 con 0x7f325c15bed0 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f3262839700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f325c0d2160 0x7f325c0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f3262839700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f325c0d2160 0x7f325c0d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35516/0 (socket says 172.21.15.77:35516) 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f3261837700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f325c1603c0 0x7f325c0d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55142/0 (socket says 172.21.15.77:55142) 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.144+0000 7f3262839700 1 -- 172.21.15.77:0/3153354771 learned_addr learned my addr 172.21.15.77:0/3153354771 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3262038700 1 --2- 172.21.15.77:0/3153354771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3262839700 1 -- 172.21.15.77:0/3153354771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 msgr2=0x7f325c0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3262839700 1 --2- 172.21.15.77:0/3153354771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3262839700 1 -- 172.21.15.77:0/3153354771 >> v1:172.21.15.77:6789/0 conn(0x7f325c1603c0 legacy=0x7f325c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3261036700 1 -- 172.21.15.77:0/3153354771 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350407287 0 0) 0x7f325c0d0770 con 0x7f325c1603c0 2024-01-15T14:34:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3262839700 1 -- 172.21.15.77:0/3153354771 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0d09a0 con 0x7f325c0d2160 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3262839700 1 --2- 172.21.15.77:0/3153354771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0d0230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32540098f0 tx=0x7f3254009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e926533b81af42fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3261036700 1 -- 172.21.15.77:0/3153354771 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254011830 con 0x7f325c0d2160 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3261036700 1 -- 172.21.15.77:0/3153354771 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254011990 con 0x7f325c0d2160 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.145+0000 7f3261036700 1 -- 172.21.15.77:0/3153354771 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254011d20 con 0x7f325c0d2160 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 -- 172.21.15.77:0/3153354771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 msgr2=0x7f325c0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 --2- 172.21.15.77:0/3153354771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0d0230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32540098f0 tx=0x7f3254009c00 comp rx=0 tx=0).stop 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 -- 172.21.15.77:0/3153354771 shutdown_connections 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 --2- 172.21.15.77:0/3153354771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0d0230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 --2- 172.21.15.77:0/3153354771 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 -- 172.21.15.77:0/3153354771 >> 172.21.15.77:0/3153354771 conn(0x7f325c073130 msgr2=0x7f325c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 -- 172.21.15.77:0/3153354771 shutdown_connections 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.146+0000 7f326429c700 1 -- 172.21.15.77:0/3153354771 wait complete. 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.147+0000 7f326429c700 1 Processor -- start 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.147+0000 7f326429c700 1 -- start start 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f325c1603c0 0x7f325c158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c146ba0 con 0x7f325c0d2160 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0c7790 con 0x7f325c1603c0 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0c7a00 con 0x7f325c15bed0 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35528/0 (socket says 172.21.15.77:35528) 2024-01-15T14:34:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 -- 172.21.15.77:0/864620709 learned_addr learned my addr 172.21.15.77:0/864620709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3261837700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262839700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f325c1603c0 0x7f325c158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 -- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 msgr2=0x7f325c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 -- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f325c1603c0 msgr2=0x7f325c158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f325c1603c0 0x7f325c158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 -- 172.21.15.77:0/864620709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3254009550 con 0x7f325c0d2160 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262839700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f325c1603c0 0x7f325c158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f3262038700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0cba40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f324c00efc0 tx=0x7f324c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00cdc0 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c00f040 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c0146a0 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.148+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c0c7ce0 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.149+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c149d40 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.149+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f324c014b00 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.149+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f324c014db0 con 0x7f325c0d2160 2024-01-15T14:34:42.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.149+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c0c8200 con 0x7f325c0d2160 2024-01-15T14:34:42.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.152+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f324c03bce0 con 0x7f325c0d2160 2024-01-15T14:34:42.217 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:42.216+0000 7f95e115d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-15T14:34:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.222+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f325c067f60 con 0x7f325c0d2160 2024-01-15T14:34:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.223+0000 7f325affd700 1 -- 172.21.15.77:0/864620709 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f324c02a070 con 0x7f325c0d2160 2024-01-15T14:34:42.224 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:42.224 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:34:41.936699+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:34:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 msgr2=0x7f325c0cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0cba40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f324c00efc0 tx=0x7f324c00c3a0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 shutdown_connections 2024-01-15T14:34:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f325c1603c0 0x7f325c158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f325c0d2160 0x7f325c0cba40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 --2- 172.21.15.77:0/864620709 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f325c15bed0 0x7f325c0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 >> 172.21.15.77:0/864620709 conn(0x7f325c073130 msgr2=0x7f325c0af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 shutdown_connections 2024-01-15T14:34:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:42.224+0000 7f326429c700 1 -- 172.21.15.77:0/864620709 wait complete. 2024-01-15T14:34:42.237 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-15T14:34:42.237 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-15T14:34:42.295 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:42.295+0000 7f95e115d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T14:34:42.576 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-01-15T14:34:42.576+0000 7f727301e640 -1 Falling back to public interface 2024-01-15T14:34:42.582 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-01-15T14:34:42.581+0000 7f0144bab640 -1 Falling back to public interface 2024-01-15T14:34:42.586 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-01-15T14:34:42.585+0000 7fca7bf79640 -1 Falling back to public interface 2024-01-15T14:34:42.588 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-01-15T14:34:42.587+0000 7f038afcf640 -1 Falling back to public interface 2024-01-15T14:34:42.588 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-01-15T14:34:42.587+0000 7f811dd76640 -1 Falling back to public interface 2024-01-15T14:34:42.588 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-01-15T14:34:42.587+0000 7f038238a700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-01-15T14:34:42.591 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-01-15T14:34:42.590+0000 7f8c927e3640 -1 Falling back to public interface 2024-01-15T14:34:42.606 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-01-15T14:34:42.605+0000 7f4b18a0a640 -1 Falling back to public interface 2024-01-15T14:34:42.606 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-01-15T14:34:42.606+0000 7f90985da640 -1 Falling back to public interface 2024-01-15T14:34:42.621 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:42.620+0000 7fec277c6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T14:34:42.702 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:42.701+0000 7fec277c6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T14:34:42.834 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:42.833+0000 7fec277c6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T14:34:43.002 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:43.002+0000 7f95e115d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T14:34:43.041 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T14:34:43.082 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:43.082+0000 7f95e115d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T14:34:43.158 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.158+0000 7f23ce6a4700 1 Processor -- start 2024-01-15T14:34:43.159 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.158+0000 7f23ce6a4700 1 -- start start 2024-01-15T14:34:43.160 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.158+0000 7f23ce6a4700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f23c815bf00 0x7f23c80aea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:43.160 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.158+0000 7f23ce6a4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80cc590 con 0x7f23c80d0450 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.158+0000 7f23ce6a4700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c815c2e0 con 0x7f23c80d2210 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23ce6a4700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c80af0a0 con 0x7f23c815bf00 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c77fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f23c80d0450 0x7f23c80d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55144/0 (socket says 172.21.15.77:55144) 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c77fe700 1 -- 172.21.15.77:0/2457323155 learned_addr learned my addr 172.21.15.77:0/2457323155 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c7fff700 1 --2- 172.21.15.77:0/2457323155 >> v2:172.21.15.159:3300/0 conn(0x7f23c815bf00 0x7f23c80aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c6ffd700 1 -- 172.21.15.77:0/2457323155 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435791925 0 0) 0x7f23c80cc590 con 0x7f23c80d0450 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c6ffd700 1 -- 172.21.15.77:0/2457323155 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b00035b0 con 0x7f23c80d0450 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c7fff700 1 -- 172.21.15.77:0/2457323155 >> v1:172.21.15.159:6789/0 conn(0x7f23c80d2210 legacy=0x7f23c80b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c7fff700 1 -- 172.21.15.77:0/2457323155 >> v1:172.21.15.77:6789/0 conn(0x7f23c80d0450 legacy=0x7f23c80d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c7fff700 1 -- 172.21.15.77:0/2457323155 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80af380 con 0x7f23c815bf00 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c7fff700 1 --2- 172.21.15.77:0/2457323155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c815bf00 0x7f23c80aea10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009840 tx=0x7f23b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426ad9095ff3a253 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c6ffd700 1 -- 172.21.15.77:0/2457323155 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8011830 con 0x7f23c815bf00 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c6ffd700 1 -- 172.21.15.77:0/2457323155 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8011990 con 0x7f23c815bf00 2024-01-15T14:34:43.161 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.159+0000 7f23c6ffd700 1 -- 172.21.15.77:0/2457323155 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8011d20 con 0x7f23c815bf00 2024-01-15T14:34:43.162 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 -- 172.21.15.77:0/2457323155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c815bf00 msgr2=0x7f23c80aea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 --2- 172.21.15.77:0/2457323155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c815bf00 0x7f23c80aea10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009840 tx=0x7f23b8009b50 comp rx=0 tx=0).stop 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 -- 172.21.15.77:0/2457323155 shutdown_connections 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 --2- 172.21.15.77:0/2457323155 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c815bf00 0x7f23c80aea10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 -- 172.21.15.77:0/2457323155 >> 172.21.15.77:0/2457323155 conn(0x7f23c809aa00 msgr2=0x7f23c809ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 -- 172.21.15.77:0/2457323155 shutdown_connections 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.160+0000 7f23ce6a4700 1 -- 172.21.15.77:0/2457323155 wait complete. 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.161+0000 7f23ce6a4700 1 Processor -- start 2024-01-15T14:34:43.163 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.161+0000 7f23ce6a4700 1 -- start start 2024-01-15T14:34:43.165 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ce6a4700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c80d0450 0x7f23c80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:43.165 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ce6a4700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23c80d2210 0x7f23c80c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:43.165 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ce6a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 0x7f23c815d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:43.165 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ce6a4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80c5770 con 0x7f23c815bf00 2024-01-15T14:34:43.165 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ce6a4700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80c59c0 con 0x7f23c80d0450 2024-01-15T14:34:43.165 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ce6a4700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80c5ca0 con 0x7f23c80d2210 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ccc41700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 0x7f23c815d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ccc41700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 0x7f23c815d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35534/0 (socket says 172.21.15.77:35534) 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23ccc41700 1 -- 172.21.15.77:0/1631829732 learned_addr learned my addr 172.21.15.77:0/1631829732 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23c7fff700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c80d0450 0x7f23c80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.162+0000 7f23c77fe700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23c80d2210 0x7f23c80c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ccc41700 1 -- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23c80d2210 msgr2=0x7f23c80c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ccc41700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23c80d2210 0x7f23c80c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ccc41700 1 -- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c80d0450 msgr2=0x7f23c80b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ccc41700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c80d0450 0x7f23c80b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ccc41700 1 -- 172.21.15.77:0/1631829732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b80094a0 con 0x7f23c815bf00 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ccc41700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 0x7f23c815d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f23c000cc50 tx=0x7f23c000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0003c00 con 0x7f23c815bf00 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c0003d60 con 0x7f23c815bf00 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c000aab0 con 0x7f23c815bf00 2024-01-15T14:34:43.166 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c80c5f80 con 0x7f23c815bf00 2024-01-15T14:34:43.167 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c80b1d70 con 0x7f23c815bf00 2024-01-15T14:34:43.167 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f23c00050d0 con 0x7f23c815bf00 2024-01-15T14:34:43.167 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.163+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f23c0012cc0 con 0x7f23c815bf00 2024-01-15T14:34:43.167 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.164+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c80c64a0 con 0x7f23c815bf00 2024-01-15T14:34:43.167 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.166+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f23c80c64a0 con 0x7f23c815bf00 2024-01-15T14:34:43.217 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:43.217+0000 7f95e115d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T14:34:43.238 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.237+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c80b1f30 con 0x7f23c815bf00 2024-01-15T14:34:43.239 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.238+0000 7f23c4ff9700 1 -- 172.21.15.77:0/1631829732 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f23c0005560 con 0x7f23c815bf00 2024-01-15T14:34:43.239 INFO:teuthology.misc.health.smithi077.stdout: 2024-01-15T14:34:43.239 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":10,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:34:41.936699+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 msgr2=0x7f23c815d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 0x7f23c815d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f23c000cc50 tx=0x7f23c000cf60 comp rx=0 tx=0).stop 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 shutdown_connections 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23c815bf00 0x7f23c815d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23c80d0450 0x7f23c80b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 --2- 172.21.15.77:0/1631829732 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23c80d2210 0x7f23c80c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 >> 172.21.15.77:0/1631829732 conn(0x7f23c809aa00 msgr2=0x7f23c809ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:43.240 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 shutdown_connections 2024-01-15T14:34:43.241 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:43.239+0000 7f23ce6a4700 1 -- 172.21.15.77:0/1631829732 wait complete. 2024-01-15T14:34:43.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-15T14:34:43.395 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:43.394+0000 7fec277c6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-15T14:34:43.566 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:43.566+0000 7fec277c6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T14:34:43.731 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:43.730+0000 7fec277c6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T14:34:43.777 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:43.777+0000 7f95e115d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-15T14:34:43.936 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:43.935+0000 7f95e115d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T14:34:44.103 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:44.103+0000 7f95e115d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T14:34:44.543 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:44.542+0000 7fec277c6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T14:34:44.766 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:44.765+0000 7fec277c6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T14:34:44.839 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:44.838+0000 7fec277c6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T14:34:44.923 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:44.923+0000 7f95e115d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T14:34:44.996 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:44.996+0000 7fec277c6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T14:34:45.164 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:45.163+0000 7f95e115d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T14:34:45.237 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:45.236+0000 7f95e115d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T14:34:45.439 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-01-15T14:34:45.438+0000 7f95e115d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T14:34:46.020 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:46.019+0000 7febe2b7e700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-15T14:34:46.020 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:34:46.020+0000 7febe2b7e700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-15T14:34:46.090 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-01-15T14:34:46.090+0000 7fca7bf79640 -1 osd.6 0 log_to_monitors true 2024-01-15T14:34:46.090 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-01-15T14:34:46.090+0000 7f727301e640 -1 osd.4 0 log_to_monitors true 2024-01-15T14:34:46.106 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-01-15T14:34:46.105+0000 7f8c927e3640 -1 osd.0 0 log_to_monitors true 2024-01-15T14:34:46.112 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-01-15T14:34:46.111+0000 7f038afcf640 -1 osd.1 0 log_to_monitors true 2024-01-15T14:34:46.113 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-01-15T14:34:46.113+0000 7f0144bab640 -1 osd.2 0 log_to_monitors true 2024-01-15T14:34:46.116 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-01-15T14:34:46.115+0000 7f811dd76640 -1 osd.3 0 log_to_monitors true 2024-01-15T14:34:46.130 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-01-15T14:34:46.130+0000 7f90985da640 -1 osd.5 0 log_to_monitors true 2024-01-15T14:34:46.130 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-01-15T14:34:46.130+0000 7f4b18a0a640 -1 osd.7 0 log_to_monitors true 2024-01-15T14:34:50.055 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T14:34:50.091 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-01-15T14:34:50.090+0000 7f4b0c5be700 -1 osd.7 0 waiting for initial osdmap 2024-01-15T14:34:50.106 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-01-15T14:34:50.104+0000 7f908c18e700 -1 osd.5 0 waiting for initial osdmap 2024-01-15T14:34:50.107 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-01-15T14:34:50.106+0000 7f4b0499e700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-01-15T14:34:50.119+0000 7f013875f700 -1 osd.2 0 waiting for initial osdmap 2024-01-15T14:34:50.124 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-01-15T14:34:50.123+0000 7f037eb83700 -1 osd.1 0 waiting for initial osdmap 2024-01-15T14:34:50.131 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-01-15T14:34:50.130+0000 7f908456e700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.131 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-01-15T14:34:50.131+0000 7f811192a700 -1 osd.3 0 waiting for initial osdmap 2024-01-15T14:34:50.132 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-01-15T14:34:50.131+0000 7f7266bd2700 -1 osd.4 0 waiting for initial osdmap 2024-01-15T14:34:50.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-01-15T14:34:50.134+0000 7f0130b3f700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.136 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-01-15T14:34:50.135+0000 7f8c86397700 -1 osd.0 0 waiting for initial osdmap 2024-01-15T14:34:50.140 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-01-15T14:34:50.139+0000 7fca6fb2d700 -1 osd.6 0 waiting for initial osdmap 2024-01-15T14:34:50.143 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-01-15T14:34:50.143+0000 7f8109d0a700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.145 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-01-15T14:34:50.145+0000 7f0376f63700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.150 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-01-15T14:34:50.149+0000 7f725efb2700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.151 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-01-15T14:34:50.151+0000 7f8c7e777700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.154 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-01-15T14:34:50.153+0000 7fca67f0d700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T14:34:50.198 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.197+0000 7efe240ff700 1 Processor -- start 2024-01-15T14:34:50.198 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.197+0000 7efe240ff700 1 -- start start 2024-01-15T14:34:50.202 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.198+0000 7efe240ff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe1c15c0b0 0x7efe1c0ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.202 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.198+0000 7efe240ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0b12f0 con 0x7efe1c0cc5e0 2024-01-15T14:34:50.202 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.198+0000 7efe240ff700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0af240 con 0x7efe1c0d06d0 2024-01-15T14:34:50.202 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.198+0000 7efe240ff700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c0af520 con 0x7efe1c15c0b0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.198+0000 7efe2169a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efe1c0cc5e0 0x7efe1c0cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35124/0 (socket says 172.21.15.77:35124) 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.198+0000 7efe2169a700 1 -- 172.21.15.77:0/1544917310 learned_addr learned my addr 172.21.15.77:0/1544917310 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe21e9b700 1 --2- 172.21.15.77:0/1544917310 >> v2:172.21.15.159:3300/0 conn(0x7efe1c15c0b0 0x7efe1c0ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191810537 0 0) 0x7efe1c0b12f0 con 0x7efe1c0cc5e0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe080035b0 con 0x7efe1c0cc5e0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485716619 0 0) 0x7efe080035b0 con 0x7efe1c0cc5e0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 >> v2:172.21.15.159:3300/0 conn(0x7efe1c15c0b0 msgr2=0x7efe1c0ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 --2- 172.21.15.77:0/1544917310 >> v2:172.21.15.159:3300/0 conn(0x7efe1c15c0b0 0x7efe1c0ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 >> v1:172.21.15.159:6789/0 conn(0x7efe1c0d06d0 legacy=0x7efe1c0b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0af800 con 0x7efe1c0cc5e0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7efe18001670 con 0x7efe1c0cc5e0 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 >> v1:172.21.15.77:6789/0 conn(0x7efe1c0cc5e0 legacy=0x7efe1c0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe08003a00 0x7efe08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe08004850 0x7efe08004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.203 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08005210 0x7efe0800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0af520 con 0x7efe08004850 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0800c1b0 con 0x7efe08003a00 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0800c370 con 0x7efe08005210 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe2169a700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe08004850 0x7efe08004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe21e9b700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe08003a00 0x7efe08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe2269c700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08005210 0x7efe0800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe2169a700 1 -- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08005210 msgr2=0x7efe0800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe2169a700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08005210 0x7efe0800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe2169a700 1 -- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe08003a00 msgr2=0x7efe08003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.199+0000 7efe2169a700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe08003a00 0x7efe08003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe2169a700 1 -- 172.21.15.77:0/1544917310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080035b0 con 0x7efe08004850 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe2269c700 1 -- 172.21.15.77:0/1544917310 reap_dead start 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe2169a700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe08004850 0x7efe08004cd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efe1800c240 tx=0x7efe08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1f65cae62d0d7a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18007c10 con 0x7efe08004850 2024-01-15T14:34:50.204 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1800cde0 con 0x7efe08004850 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe20e99700 1 -- 172.21.15.77:0/1544917310 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18005ba0 con 0x7efe08004850 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 -- 172.21.15.77:0/1544917310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe08004850 msgr2=0x7efe08004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe08004850 0x7efe08004cd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efe1800c240 tx=0x7efe08005880 comp rx=0 tx=0).stop 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 -- 172.21.15.77:0/1544917310 shutdown_connections 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 --2- 172.21.15.77:0/1544917310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe08004850 0x7efe08004cd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 -- 172.21.15.77:0/1544917310 >> 172.21.15.77:0/1544917310 conn(0x7efe1c09aa00 msgr2=0x7efe1c09ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 -- 172.21.15.77:0/1544917310 shutdown_connections 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.200+0000 7efe240ff700 1 -- 172.21.15.77:0/1544917310 wait complete. 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 Processor -- start 2024-01-15T14:34:50.205 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 -- start start 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08004850 0x7efe1c1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe1c150e10 0x7efe1c151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 0x7efe1c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c15a380 con 0x7efe1c1546a0 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c159a90 con 0x7efe1c150e10 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe240ff700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c159d30 con 0x7efe08004850 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe2269c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 0x7efe1c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe2269c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 0x7efe1c15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39268/0 (socket says 172.21.15.77:39268) 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.202+0000 7efe2269c700 1 -- 172.21.15.77:0/448307657 learned_addr learned my addr 172.21.15.77:0/448307657 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2169a700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe1c150e10 0x7efe1c151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe21e9b700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08004850 0x7efe1c1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2269c700 1 -- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08004850 msgr2=0x7efe1c1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.208 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2269c700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08004850 0x7efe1c1559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2269c700 1 -- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe1c150e10 msgr2=0x7efe1c151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2269c700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe1c150e10 0x7efe1c151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2269c700 1 -- 172.21.15.77:0/448307657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c15a010 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe2269c700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 0x7efe1c15d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0728a0 tx=0x7efe1400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14003c10 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14003d70 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c155730 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14005310 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.203+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c154b20 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.204+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c068fb0 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.204+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7efe14005470 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.205+0000 7efe127fc700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efe0006b2f0 0x7efe0006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.205+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7efe14088c00 con 0x7efe1c1546a0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.205+0000 7efe21e9b700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efe0006b2f0 0x7efe0006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:50.209 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.205+0000 7efe21e9b700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efe0006b2f0 0x7efe0006d7b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009c20 tx=0x7efe0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:50.210 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.210+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe14054970 con 0x7efe1c1546a0 2024-01-15T14:34:50.403 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.402+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe1c15a4c0 con 0x7efe1c1546a0 2024-01-15T14:34:50.404 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.403+0000 7efe127fc700 1 -- 172.21.15.77:0/448307657 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7efe1c15a4c0 con 0x7efe1c1546a0 2024-01-15T14:34:50.404 INFO:teuthology.misc.health.smithi077.stdout: 2024-01-15T14:34:50.404 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":12,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:34:48.073732+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:34:50.406 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efe0006b2f0 msgr2=0x7efe0006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.406 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efe0006b2f0 0x7efe0006d7b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009c20 tx=0x7efe0c008040 comp rx=0 tx=0).stop 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 msgr2=0x7efe1c15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 0x7efe1c15d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0728a0 tx=0x7efe1400ee00 comp rx=0 tx=0).stop 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 shutdown_connections 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efe0006b2f0 0x7efe0006d7b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe08004850 0x7efe1c1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe1c1546a0 0x7efe1c15d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 --2- 172.21.15.77:0/448307657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe1c150e10 0x7efe1c151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 >> 172.21.15.77:0/448307657 conn(0x7efe1c09aa00 msgr2=0x7efe1c0b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 shutdown_connections 2024-01-15T14:34:50.407 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:50.405+0000 7efe240ff700 1 -- 172.21.15.77:0/448307657 wait complete. 2024-01-15T14:34:50.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-15T14:34:57.225 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T14:34:57.350 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.349+0000 7fa3dd361700 1 Processor -- start 2024-01-15T14:34:57.350 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.349+0000 7fa3dd361700 1 -- start start 2024-01-15T14:34:57.359 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3dd361700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa3d8161610 0x7fa3d80d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.359 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3dd361700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d80b9940 con 0x7fa3d81619f0 2024-01-15T14:34:57.359 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3dd361700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d80b9c20 con 0x7fa3d806b700 2024-01-15T14:34:57.359 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3dd361700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d80b9f00 con 0x7fa3d8161610 2024-01-15T14:34:57.359 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d67fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa3d81619f0 0x7fa3d806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35130/0 (socket says 172.21.15.77:35130) 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d67fc700 1 -- 172.21.15.77:0/927031255 learned_addr learned my addr 172.21.15.77:0/927031255 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508940861 0 0) 0x7fa3d80b9940 con 0x7fa3d81619f0 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c00035b0 con 0x7fa3d81619f0 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/927031255 >> v2:172.21.15.159:3300/0 conn(0x7fa3d8161610 0x7fa3d80d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578265548 0 0) 0x7fa3c00035b0 con 0x7fa3d81619f0 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 >> v2:172.21.15.159:3300/0 conn(0x7fa3d8161610 msgr2=0x7fa3d80d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 --2- 172.21.15.77:0/927031255 >> v2:172.21.15.159:3300/0 conn(0x7fa3d8161610 0x7fa3d80d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 >> v1:172.21.15.159:6789/0 conn(0x7fa3d806b700 legacy=0x7fa3d80d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.360 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80d1b80 con 0x7fa3d81619f0 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fa3c80017f0 con 0x7fa3d81619f0 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/927031255 >> v2:172.21.15.159:3300/0 conn(0x7fa3d8161610 0x7fa3d80d3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 >> v1:172.21.15.77:6789/0 conn(0x7fa3d81619f0 legacy=0x7fa3d806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3c0003a00 0x7fa3c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0004850 0x7fa3c0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3c0005210 0x7fa3c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80b9f00 con 0x7fa3c0003a00 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c000c140 con 0x7fa3c0004850 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c000c300 con 0x7fa3c0005210 2024-01-15T14:34:57.361 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.350+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3c0003a00 0x7fa3c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d67fc700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0004850 0x7fa3c0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d77fe700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3c0005210 0x7fa3c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d6ffd700 1 -- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3c0005210 msgr2=0x7fa3c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3c0005210 0x7fa3c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d6ffd700 1 -- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0004850 msgr2=0x7fa3c0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0004850 0x7fa3c0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d6ffd700 1 -- 172.21.15.77:0/927031255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00035b0 con 0x7fa3c0003a00 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d77fe700 1 -- 172.21.15.77:0/927031255 reap_dead start 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3c0003a00 0x7fa3c0003e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009950 tx=0x7fa3c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6643f3dbd56f8117 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:57.362 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00c070 con 0x7fa3c0003a00 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc013610 con 0x7fa3c0003a00 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3d5ffb700 1 -- 172.21.15.77:0/927031255 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc0139a0 con 0x7fa3c0003a00 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.351+0000 7fa3dd361700 1 -- 172.21.15.77:0/927031255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3c0003a00 msgr2=0x7fa3c0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.352+0000 7fa3dd361700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3c0003a00 0x7fa3c0003e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009950 tx=0x7fa3c0005880 comp rx=0 tx=0).stop 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.352+0000 7fa3dd361700 1 -- 172.21.15.77:0/927031255 shutdown_connections 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.352+0000 7fa3dd361700 1 --2- 172.21.15.77:0/927031255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3c0003a00 0x7fa3c0003e20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.352+0000 7fa3dd361700 1 -- 172.21.15.77:0/927031255 >> 172.21.15.77:0/927031255 conn(0x7fa3d80d44d0 msgr2=0x7fa3d80d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.352+0000 7fa3dd361700 1 -- 172.21.15.77:0/927031255 shutdown_connections 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.352+0000 7fa3dd361700 1 -- 172.21.15.77:0/927031255 wait complete. 2024-01-15T14:34:57.363 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.353+0000 7fa3dd361700 1 Processor -- start 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.353+0000 7fa3dd361700 1 -- start start 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0003a00 0x7fa3d81511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 0x7fa3d80cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80ce0d0 0x7fa3d80ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80c7940 con 0x7fa3d80caf70 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80cc770 con 0x7fa3c0003a00 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80cca50 con 0x7fa3d80ce0d0 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 0x7fa3d80cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 0x7fa3d80cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39314/0 (socket says 172.21.15.77:39314) 2024-01-15T14:34:57.364 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 -- 172.21.15.77:0/1407786592 learned_addr learned my addr 172.21.15.77:0/1407786592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d77fe700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80ce0d0 0x7fa3d80ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0003a00 0x7fa3d81511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 -- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80ce0d0 msgr2=0x7fa3d80ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80ce0d0 0x7fa3d80ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 -- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0003a00 msgr2=0x7fa3d81511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0003a00 0x7fa3d81511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 -- 172.21.15.77:0/1407786592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80c7b90 con 0x7fa3d80caf70 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3d67fc700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 0x7fa3d80cbdd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8001920 tx=0x7fa3c800fc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80070d0 con 0x7fa3d80caf70 2024-01-15T14:34:57.365 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8007250 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d80c7e70 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8007600 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.354+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d8151720 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.355+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d80c80b0 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.356+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa3c8005a40 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.356+0000 7fa3bf7fe700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa3c406b350 0x7fa3c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.356+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa3c8089910 con 0x7fa3d80caf70 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.356+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa3c406b350 0x7fa3c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.366 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.357+0000 7fa3d6ffd700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa3c406b350 0x7fa3c406d810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc000c00 tx=0x7fa3cc01e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:57.367 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.361+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3d80c80b0 con 0x7fa3d80caf70 2024-01-15T14:34:57.554 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.553+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3d80c80b0 con 0x7fa3d80caf70 2024-01-15T14:34:57.555 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.554+0000 7fa3bf7fe700 1 -- 172.21.15.77:0/1407786592 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fa3d80c80b0 con 0x7fa3d80caf70 2024-01-15T14:34:57.555 INFO:teuthology.misc.health.smithi077.stdout: 2024-01-15T14:34:57.556 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":17,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:34:55.161631+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa3c406b350 msgr2=0x7fa3c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa3c406b350 0x7fa3c406d810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc000c00 tx=0x7fa3cc01e040 comp rx=0 tx=0).stop 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 msgr2=0x7fa3d80cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 0x7fa3d80cbdd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8001920 tx=0x7fa3c800fc40 comp rx=0 tx=0).stop 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 shutdown_connections 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa3c406b350 0x7fa3c406d810 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa3d80ce0d0 0x7fa3d80ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa3c0003a00 0x7fa3d81511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.558 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 --2- 172.21.15.77:0/1407786592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80caf70 0x7fa3d80cbdd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.559 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 >> 172.21.15.77:0/1407786592 conn(0x7fa3d80d44d0 msgr2=0x7fa3d80b7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:57.559 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 shutdown_connections 2024-01-15T14:34:57.559 INFO:teuthology.misc.health.smithi077.stderr:2024-01-15T14:34:57.556+0000 7fa3dd361700 1 -- 172.21.15.77:0/1407786592 wait complete. 2024-01-15T14:34:57.568 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-15T14:34:57.569 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-15T14:34:57.569 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T14:34:57.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T14:34:57.572 INFO:tasks.ceph.mds.a:Started 2024-01-15T14:34:57.572 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T14:34:57.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T14:34:57.575 INFO:tasks.ceph.mds.c:Started 2024-01-15T14:34:57.575 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T14:34:57.575 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T14:34:57.579 INFO:tasks.ceph.mds.b:Started 2024-01-15T14:34:57.579 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T14:34:57.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T14:34:57.582 INFO:tasks.ceph.mds.d:Started 2024-01-15T14:34:57.583 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-15T14:34:57.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:34:57.652 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-01-15T14:34:57.653 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-01-15T14:34:57.653 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-01-15T14:34:57.652+0000 7efe4d1bc700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-01-15T14:34:57.670 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-01-15T14:34:57.670 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-01-15T14:34:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.711+0000 7fb676d10700 1 Processor -- start 2024-01-15T14:34:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.711+0000 7fb676d10700 1 -- start start 2024-01-15T14:34:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb676d10700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb6700b0ba0 0x7fb6700af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb676d10700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67015bb60 con 0x7fb67015ee90 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb676d10700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700b2bd0 con 0x7fb6700b0f80 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb676d10700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6700b1fc0 con 0x7fb6700b0ba0 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb6752ad700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb67015ee90 0x7fb67015b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35134/0 (socket says 172.21.15.77:35134) 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb6752ad700 1 -- 172.21.15.77:0/173288429 learned_addr learned my addr 172.21.15.77:0/173288429 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb674aac700 1 --2- 172.21.15.77:0/173288429 >> v2:172.21.15.159:3300/0 conn(0x7fb6700b0ba0 0x7fb6700af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb66f7fe700 1 -- 172.21.15.77:0/173288429 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747494636 0 0) 0x7fb668001490 con 0x7fb67015ee90 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb66f7fe700 1 -- 172.21.15.77:0/173288429 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6600046c0 con 0x7fb67015ee90 2024-01-15T14:34:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb674aac700 1 -- 172.21.15.77:0/173288429 >> v1:172.21.15.159:6789/0 conn(0x7fb6700b0f80 legacy=0x7fb6700cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.712+0000 7fb674aac700 1 -- 172.21.15.77:0/173288429 >> v1:172.21.15.77:6789/0 conn(0x7fb67015ee90 legacy=0x7fb67015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb674aac700 1 -- 172.21.15.77:0/173288429 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700b22a0 con 0x7fb6700b0ba0 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb674aac700 1 --2- 172.21.15.77:0/173288429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0ba0 0x7fb6700af5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb664009840 tx=0x7fb664009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89dd1a65fe6e2c71 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb66f7fe700 1 -- 172.21.15.77:0/173288429 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb664011830 con 0x7fb6700b0ba0 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb66f7fe700 1 -- 172.21.15.77:0/173288429 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb664011990 con 0x7fb6700b0ba0 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb66f7fe700 1 -- 172.21.15.77:0/173288429 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb664011d20 con 0x7fb6700b0ba0 2024-01-15T14:34:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 -- 172.21.15.77:0/173288429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0ba0 msgr2=0x7fb6700af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 --2- 172.21.15.77:0/173288429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0ba0 0x7fb6700af5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb664009840 tx=0x7fb664009b50 comp rx=0 tx=0).stop 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 -- 172.21.15.77:0/173288429 shutdown_connections 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 --2- 172.21.15.77:0/173288429 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0ba0 0x7fb6700af5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 -- 172.21.15.77:0/173288429 >> 172.21.15.77:0/173288429 conn(0x7fb6700b57b0 msgr2=0x7fb6700b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 -- 172.21.15.77:0/173288429 shutdown_connections 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.713+0000 7fb676d10700 1 -- 172.21.15.77:0/173288429 wait complete. 2024-01-15T14:34:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.715+0000 7fb676d10700 1 Processor -- start 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.715+0000 7fb676d10700 1 -- start start 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.715+0000 7fb676d10700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6700b0ba0 0x7fb6700af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.715+0000 7fb676d10700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0f80 0x7fb670154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.715+0000 7fb676d10700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 0x7fb67006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb676d10700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700aec30 con 0x7fb67015ee90 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb676d10700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700ae090 con 0x7fb6700b0f80 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb676d10700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700ae300 con 0x7fb6700b0ba0 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 0x7fb67006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 0x7fb67006e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39346/0 (socket says 172.21.15.77:39346) 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 -- 172.21.15.77:0/916477826 learned_addr learned my addr 172.21.15.77:0/916477826 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb674aac700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6700b0ba0 0x7fb6700af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb66ffff700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0f80 0x7fb670154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 -- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6700b0ba0 msgr2=0x7fb6700af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6700b0ba0 0x7fb6700af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 -- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0f80 msgr2=0x7fb670154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0f80 0x7fb670154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 -- 172.21.15.77:0/916477826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6640094a0 con 0x7fb67015ee90 2024-01-15T14:34:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb66ffff700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0f80 0x7fb670154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb6752ad700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 0x7fb67006e800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb66800cb10 tx=0x7fb66800ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66800aa20 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66800ab80 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6700ae5e0 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668012400 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.716+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6700af600 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.717+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6700aede0 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.718+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb66801b3b0 con 0x7fb67015ee90 2024-01-15T14:34:57.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.718+0000 7fb66d7fa700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb65406b350 0x7fb65406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.718+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb668089350 con 0x7fb67015ee90 2024-01-15T14:34:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.718+0000 7fb674aac700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb65406b350 0x7fb65406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.719+0000 7fb674aac700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb65406b350 0x7fb65406d810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb6640090b0 tx=0x7fb664008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.723+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6700aede0 con 0x7fb67015ee90 2024-01-15T14:34:57.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.913+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6700aede0 con 0x7fb67015ee90 2024-01-15T14:34:57.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.913+0000 7fb66d7fa700 1 -- 172.21.15.77:0/916477826 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fb6700aede0 con 0x7fb67015ee90 2024-01-15T14:34:57.915 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:57.915 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:34:55.161631+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:34:57.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb65406b350 msgr2=0x7fb65406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb65406b350 0x7fb65406d810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb6640090b0 tx=0x7fb664008030 comp rx=0 tx=0).stop 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 msgr2=0x7fb67006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 0x7fb67006e800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb66800cb10 tx=0x7fb66800ce20 comp rx=0 tx=0).stop 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 shutdown_connections 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb65406b350 0x7fb65406d810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6700b0ba0 0x7fb6700af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67015ee90 0x7fb67006e800 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 --2- 172.21.15.77:0/916477826 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6700b0f80 0x7fb670154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 >> 172.21.15.77:0/916477826 conn(0x7fb6700b57b0 msgr2=0x7fb6700cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:57.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 shutdown_connections 2024-01-15T14:34:57.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:57.916+0000 7fb676d10700 1 -- 172.21.15.77:0/916477826 wait complete. 2024-01-15T14:34:57.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:34:57.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:34:58.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.052+0000 7f8d5ec40700 1 Processor -- start 2024-01-15T14:34:58.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.052+0000 7f8d5ec40700 1 -- start start 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5ec40700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d5815ee20 0x7f8d5806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5ec40700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d580b5600 con 0x7f8d5806f580 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5ec40700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5806fa10 con 0x7f8d58158600 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5ec40700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5806e4e0 con 0x7f8d5815ee20 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5d1dd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d5815ee20 0x7f8d5806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d57fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8d5806f580 0x7f8d5815e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35144/0 (socket says 172.21.15.77:35144) 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5d1dd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d5815ee20 0x7f8d5806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39360/0 (socket says 172.21.15.77:39360) 2024-01-15T14:34:58.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d57fff700 1 -- 172.21.15.77:0/1308422793 learned_addr learned my addr 172.21.15.77:0/1308422793 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5d1dd700 1 -- 172.21.15.77:0/1308422793 >> v1:172.21.15.159:6789/0 conn(0x7f8d58158600 legacy=0x7f8d580b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d5d1dd700 1 -- 172.21.15.77:0/1308422793 >> v1:172.21.15.77:6789/0 conn(0x7f8d5806f580 legacy=0x7f8d5815e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5d1dd700 1 -- 172.21.15.77:0/1308422793 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5806e7c0 con 0x7f8d5815ee20 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.053+0000 7f8d577fe700 1 -- 172.21.15.77:0/1308422793 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874539652 0 0) 0x7f8d580b5600 con 0x7f8d5806f580 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5d1dd700 1 --2- 172.21.15.77:0/1308422793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5815ee20 0x7f8d5806aff0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009dc0 tx=0x7f8d5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5673bae6364a545 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d577fe700 1 -- 172.21.15.77:0/1308422793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000e070 con 0x7f8d5815ee20 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d577fe700 1 -- 172.21.15.77:0/1308422793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5000bb90 con 0x7f8d5815ee20 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d577fe700 1 -- 172.21.15.77:0/1308422793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50012430 con 0x7f8d5815ee20 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 -- 172.21.15.77:0/1308422793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5815ee20 msgr2=0x7f8d5806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/1308422793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5815ee20 0x7f8d5806aff0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009dc0 tx=0x7f8d5000b330 comp rx=0 tx=0).stop 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 -- 172.21.15.77:0/1308422793 shutdown_connections 2024-01-15T14:34:58.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/1308422793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5815ee20 0x7f8d5806aff0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 -- 172.21.15.77:0/1308422793 >> 172.21.15.77:0/1308422793 conn(0x7f8d580b3250 msgr2=0x7f8d580d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:58.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 -- 172.21.15.77:0/1308422793 shutdown_connections 2024-01-15T14:34:58.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.054+0000 7f8d5ec40700 1 -- 172.21.15.77:0/1308422793 wait complete. 2024-01-15T14:34:58.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 Processor -- start 2024-01-15T14:34:58.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 -- start start 2024-01-15T14:34:58.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5806f580 0x7f8d5814b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 0x7f8d580cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5815ee20 0x7f8d5815c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58149f90 con 0x7f8d58158600 2024-01-15T14:34:58.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580ce6c0 con 0x7f8d5815ee20 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.056+0000 7f8d5ec40700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580ce100 con 0x7f8d5806f580 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 0x7f8d580cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 0x7f8d580cfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39376/0 (socket says 172.21.15.77:39376) 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 -- 172.21.15.77:0/2730831200 learned_addr learned my addr 172.21.15.77:0/2730831200 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d5d1dd700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5815ee20 0x7f8d5815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d5c9dc700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5806f580 0x7f8d5814b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 -- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5806f580 msgr2=0x7f8d5814b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5806f580 0x7f8d5814b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 -- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5815ee20 msgr2=0x7f8d5815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5815ee20 0x7f8d5815c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 -- 172.21.15.77:0/2730831200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50009a70 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d57fff700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 0x7f8d580cfeb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c520 tx=0x7f8d4400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44004c80 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d44004e00 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44007310 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d580ce3e0 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.057+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d58142280 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.058+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58149a80 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.058+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8d44007470 con 0x7f8d58158600 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.059+0000 7f8d557fa700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d3c06b350 0x7f8d3c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.059+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8d44089430 con 0x7f8d58158600 2024-01-15T14:34:58.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.059+0000 7f8d5c9dc700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d3c06b350 0x7f8d3c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.059+0000 7f8d5c9dc700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d3c06b350 0x7f8d3c06d810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c007fd0 tx=0x7f8d4c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:58.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.064+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d58149a80 con 0x7f8d58158600 2024-01-15T14:34:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.254+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d58149a80 con 0x7f8d58158600 2024-01-15T14:34:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.254+0000 7f8d557fa700 1 -- 172.21.15.77:0/2730831200 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d58149a80 con 0x7f8d58158600 2024-01-15T14:34:58.256 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:34:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d3c06b350 msgr2=0x7f8d3c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d3c06b350 0x7f8d3c06d810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c007fd0 tx=0x7f8d4c009040 comp rx=0 tx=0).stop 2024-01-15T14:34:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 msgr2=0x7f8d580cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 0x7f8d580cfeb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c520 tx=0x7f8d4400c830 comp rx=0 tx=0).stop 2024-01-15T14:34:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 shutdown_connections 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d3c06b350 0x7f8d3c06d810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d5806f580 0x7f8d5814b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d5815ee20 0x7f8d5815c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 --2- 172.21.15.77:0/2730831200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d58158600 0x7f8d580cfeb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 >> 172.21.15.77:0/2730831200 conn(0x7f8d580b3250 msgr2=0x7f8d58158f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 shutdown_connections 2024-01-15T14:34:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.257+0000 7f8d5ec40700 1 -- 172.21.15.77:0/2730831200 wait complete. 2024-01-15T14:34:58.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:34:58.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.390+0000 7fe192aa7700 1 Processor -- start 2024-01-15T14:34:58.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.390+0000 7fe192aa7700 1 -- start start 2024-01-15T14:34:58.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe192aa7700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe18c15bed0 0x7fe18c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe192aa7700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe192aa7700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18c0d0770 con 0x7fe18c1603c0 2024-01-15T14:34:58.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe192aa7700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c15c2b0 con 0x7fe18c15bed0 2024-01-15T14:34:58.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe192aa7700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0cd100 con 0x7fe18c0d2160 2024-01-15T14:34:58.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe18bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe18c1603c0 0x7fe18c0d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35154/0 (socket says 172.21.15.77:35154) 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe18bfff700 1 -- 172.21.15.77:0/2639349150 learned_addr learned my addr 172.21.15.77:0/2639349150 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe191044700 1 --2- 172.21.15.77:0/2639349150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe190843700 1 --2- 172.21.15.77:0/2639349150 >> v2:172.21.15.159:3300/0 conn(0x7fe18c15bed0 0x7fe18c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe18b7fe700 1 -- 172.21.15.77:0/2639349150 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687303715 0 0) 0x7fe18c0d0770 con 0x7fe18c1603c0 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe18b7fe700 1 -- 172.21.15.77:0/2639349150 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe17c0035b0 con 0x7fe18c1603c0 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe18b7fe700 1 -- 172.21.15.77:0/2639349150 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109411200 0 0) 0x7fe17c0035b0 con 0x7fe18c1603c0 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe191044700 1 -- 172.21.15.77:0/2639349150 >> v2:172.21.15.159:3300/0 conn(0x7fe18c15bed0 msgr2=0x7fe18c0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe191044700 1 --2- 172.21.15.77:0/2639349150 >> v2:172.21.15.159:3300/0 conn(0x7fe18c15bed0 0x7fe18c0b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe191044700 1 -- 172.21.15.77:0/2639349150 >> v1:172.21.15.77:6789/0 conn(0x7fe18c1603c0 legacy=0x7fe18c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe191044700 1 -- 172.21.15.77:0/2639349150 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c0d09a0 con 0x7fe18c0d2160 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.391+0000 7fe190843700 1 --2- 172.21.15.77:0/2639349150 >> v2:172.21.15.159:3300/0 conn(0x7fe18c15bed0 0x7fe18c0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe191044700 1 --2- 172.21.15.77:0/2639349150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c0d0230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe184009d90 tx=0x7fe18400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903ff4a76cb0ce6d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe18b7fe700 1 -- 172.21.15.77:0/2639349150 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18400e070 con 0x7fe18c0d2160 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe18b7fe700 1 -- 172.21.15.77:0/2639349150 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184003a20 con 0x7fe18c0d2160 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe18b7fe700 1 -- 172.21.15.77:0/2639349150 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178001670 con 0x7fe18c0d2160 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 -- 172.21.15.77:0/2639349150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 msgr2=0x7fe18c0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 --2- 172.21.15.77:0/2639349150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c0d0230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe184009d90 tx=0x7fe18400b330 comp rx=0 tx=0).stop 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 -- 172.21.15.77:0/2639349150 shutdown_connections 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 --2- 172.21.15.77:0/2639349150 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c0d0230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 --2- 172.21.15.77:0/2639349150 >> v2:172.21.15.159:3300/0 conn(0x7fe18c15bed0 0x7fe18c0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 -- 172.21.15.77:0/2639349150 >> 172.21.15.77:0/2639349150 conn(0x7fe18c073130 msgr2=0x7fe18c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:58.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 -- 172.21.15.77:0/2639349150 shutdown_connections 2024-01-15T14:34:58.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.392+0000 7fe192aa7700 1 -- 172.21.15.77:0/2639349150 wait complete. 2024-01-15T14:34:58.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 Processor -- start 2024-01-15T14:34:58.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 -- start start 2024-01-15T14:34:58.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c15eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe18c15bed0 0x7fe18c0b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 0x7fe18c153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0b4e20 con 0x7fe18c1603c0 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c154320 con 0x7fe18c15bed0 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.394+0000 7fe192aa7700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c15f190 con 0x7fe18c0d2160 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 0x7fe18c153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 0x7fe18c153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39382/0 (socket says 172.21.15.77:39382) 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 -- 172.21.15.77:0/1774024455 learned_addr learned my addr 172.21.15.77:0/1774024455 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe18bfff700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe18c15bed0 0x7fe18c0b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe190843700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c15eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 -- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 msgr2=0x7fe18c15eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c15eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 -- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe18c15bed0 msgr2=0x7fe18c0b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe18c15bed0 0x7fe18c0b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 -- 172.21.15.77:0/1774024455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800094a0 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe190843700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c15eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe191044700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 0x7fe18c153de0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe184004000 tx=0x7fe184004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184009410 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184004590 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe184009a70 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184004940 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.395+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c0c9000 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.396+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c067f60 con 0x7fe18c1603c0 2024-01-15T14:34:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.396+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe184012450 con 0x7fe18c1603c0 2024-01-15T14:34:58.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.397+0000 7fe1897fa700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe17006b350 0x7fe17006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:34:58.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.397+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe18408b7c0 con 0x7fe18c1603c0 2024-01-15T14:34:58.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.397+0000 7fe190843700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe17006b350 0x7fe17006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:34:58.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.397+0000 7fe190843700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe17006b350 0x7fe17006d810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe180009cf0 tx=0x7fe180008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:34:58.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.401+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe18405a4a0 con 0x7fe18c1603c0 2024-01-15T14:34:58.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.634+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe18c0b4600 con 0x7fe18c1603c0 2024-01-15T14:34:58.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.634+0000 7fe1897fa700 1 -- 172.21.15.77:0/1774024455 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fe18405a030 con 0x7fe18c1603c0 2024-01-15T14:34:58.636 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:34:58.636 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-15T14:34:58.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe17006b350 msgr2=0x7fe17006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe17006b350 0x7fe17006d810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe180009cf0 tx=0x7fe180008040 comp rx=0 tx=0).stop 2024-01-15T14:34:58.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 msgr2=0x7fe18c153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 0x7fe18c153de0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe184004000 tx=0x7fe184004ad0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 shutdown_connections 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe17006b350 0x7fe17006d810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe18c0d2160 0x7fe18c15eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18c1603c0 0x7fe18c153de0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 --2- 172.21.15.77:0/1774024455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe18c15bed0 0x7fe18c0b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 >> 172.21.15.77:0/1774024455 conn(0x7fe18c073130 msgr2=0x7fe18c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 shutdown_connections 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:34:58.636+0000 7fe192aa7700 1 -- 172.21.15.77:0/1774024455 wait complete. 2024-01-15T14:34:58.639 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-01-15T14:35:00.650 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.785+0000 7f7b37cdb700 1 Processor -- start 2024-01-15T14:35:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.785+0000 7f7b37cdb700 1 -- start start 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b37cdb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b30072bb0 0x7f7b3006efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b37cdb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b300b4260 0x7f7b30070d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b37cdb700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3008bdb0 con 0x7f7b3016ae60 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b37cdb700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b3006ce30 con 0x7f7b300b4260 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b37cdb700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3006d110 con 0x7f7b30072bb0 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b36278700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b300b4260 0x7f7b30070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b35a77700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7b3016ae60 0x7f7b300b3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48430/0 (socket says 172.21.15.77:48430) 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b35a77700 1 -- 172.21.15.77:0/1181189095 learned_addr learned my addr 172.21.15.77:0/1181189095 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b34a75700 1 -- 172.21.15.77:0/1181189095 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320837748 0 0) 0x7f7b3008bdb0 con 0x7f7b3016ae60 2024-01-15T14:35:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.786+0000 7f7b35276700 1 --2- 172.21.15.77:0/1181189095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b30072bb0 0x7f7b3006efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b34a75700 1 -- 172.21.15.77:0/1181189095 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1c003590 con 0x7f7b3016ae60 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b36278700 1 -- 172.21.15.77:0/1181189095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b30072bb0 msgr2=0x7f7b3006efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b36278700 1 --2- 172.21.15.77:0/1181189095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b30072bb0 0x7f7b3006efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b36278700 1 -- 172.21.15.77:0/1181189095 >> v1:172.21.15.77:6789/0 conn(0x7f7b3016ae60 legacy=0x7f7b300b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b34a75700 1 -- 172.21.15.77:0/1181189095 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563198582 0 0) 0x7f7b1c003590 con 0x7f7b3016ae60 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b36278700 1 -- 172.21.15.77:0/1181189095 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3006d3f0 con 0x7f7b300b4260 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b36278700 1 --2- 172.21.15.77:0/1181189095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b300b4260 0x7f7b30070d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800be30 tx=0x7f7b280092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95510b19a21da693 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b34a75700 1 -- 172.21.15.77:0/1181189095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20001670 con 0x7f7b300b4260 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b34a75700 1 -- 172.21.15.77:0/1181189095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2800b2e0 con 0x7f7b300b4260 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b34a75700 1 -- 172.21.15.77:0/1181189095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28003c40 con 0x7f7b300b4260 2024-01-15T14:35:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 -- 172.21.15.77:0/1181189095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b300b4260 msgr2=0x7f7b30070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/1181189095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b300b4260 0x7f7b30070d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800be30 tx=0x7f7b280092f0 comp rx=0 tx=0).stop 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 -- 172.21.15.77:0/1181189095 shutdown_connections 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/1181189095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b300b4260 0x7f7b30070d00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/1181189095 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b30072bb0 0x7f7b3006efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 -- 172.21.15.77:0/1181189095 >> 172.21.15.77:0/1181189095 conn(0x7f7b300cfa30 msgr2=0x7f7b300cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 -- 172.21.15.77:0/1181189095 shutdown_connections 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.798+0000 7f7b37cdb700 1 -- 172.21.15.77:0/1181189095 wait complete. 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.799+0000 7f7b37cdb700 1 Processor -- start 2024-01-15T14:35:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.799+0000 7f7b37cdb700 1 -- start start 2024-01-15T14:35:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b37cdb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b300b4260 0x7f7b30204290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b37cdb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 0x7f7b3016b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b37cdb700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b300b3a10 0x7f7b3016bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b37cdb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3016cfa0 con 0x7f7b3016dcf0 2024-01-15T14:35:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b37cdb700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b302047d0 con 0x7f7b300b3a10 2024-01-15T14:35:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b37cdb700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3016d210 con 0x7f7b300b4260 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b35276700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 0x7f7b3016b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b35276700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 0x7f7b3016b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56076/0 (socket says 172.21.15.77:56076) 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b35276700 1 -- 172.21.15.77:0/4080834867 learned_addr learned my addr 172.21.15.77:0/4080834867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b35a77700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b300b4260 0x7f7b30204290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.800+0000 7f7b36278700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b300b3a10 0x7f7b3016bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b35276700 1 -- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b300b4260 msgr2=0x7f7b30204290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b35276700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b300b4260 0x7f7b30204290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b35276700 1 -- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b300b3a10 msgr2=0x7f7b3016bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b35276700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b300b3a10 0x7f7b3016bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b35276700 1 -- 172.21.15.77:0/4080834867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2800ba90 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b36278700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b300b3a10 0x7f7b3016bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b35276700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 0x7f7b3016b750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c007c30 tx=0x7f7b2c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c006cf0 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c00d040 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c013890 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3016d4f0 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.801+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b300b3fb0 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.802+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3016da10 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.802+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b2c0139f0 con 0x7f7b3016dcf0 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.803+0000 7f7b267fc700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b1406b350 0x7f7b1406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:00.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.803+0000 7f7b35a77700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b1406b350 0x7f7b1406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:00.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.803+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b2c08a030 con 0x7f7b3016dcf0 2024-01-15T14:35:00.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.803+0000 7f7b35a77700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b1406b350 0x7f7b1406d810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000f60 tx=0x7f7b20001e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:00.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:00.807+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b3016da10 con 0x7f7b3016dcf0 2024-01-15T14:35:01.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.042+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b3016da10 con 0x7f7b3016dcf0 2024-01-15T14:35:01.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.042+0000 7f7b267fc700 1 -- 172.21.15.77:0/4080834867 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7f7b2c058c90 con 0x7f7b3016dcf0 2024-01-15T14:35:01.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:01.044 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-15T14:35:01.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b1406b350 msgr2=0x7f7b1406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b1406b350 0x7f7b1406d810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000f60 tx=0x7f7b20001e00 comp rx=0 tx=0).stop 2024-01-15T14:35:01.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 msgr2=0x7f7b3016b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 0x7f7b3016b750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c007c30 tx=0x7f7b2c009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:01.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 shutdown_connections 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b1406b350 0x7f7b1406d810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b300b4260 0x7f7b30204290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b300b3a10 0x7f7b3016bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 --2- 172.21.15.77:0/4080834867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b3016dcf0 0x7f7b3016b750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 >> 172.21.15.77:0/4080834867 conn(0x7f7b300cfa30 msgr2=0x7f7b300af840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 shutdown_connections 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.044+0000 7f7b37cdb700 1 -- 172.21.15.77:0/4080834867 wait complete. 2024-01-15T14:35:01.047 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2024-01-15T14:35:01.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:01.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.180+0000 7f249aaf8700 1 Processor -- start 2024-01-15T14:35:01.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.180+0000 7f249aaf8700 1 -- start start 2024-01-15T14:35:01.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f249aaf8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249406c710 0x7f24940b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f249aaf8700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f24940af4d0 0x7f24940afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f249aaf8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ae430 con 0x7f249415e080 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f249aaf8700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24940af0b0 con 0x7f24940af4d0 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f249aaf8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940b05a0 con 0x7f249406c710 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f2498894700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f249415e080 0x7f249406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48442/0 (socket says 172.21.15.77:48442) 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f2498894700 1 -- 172.21.15.77:0/24288073 learned_addr learned my addr 172.21.15.77:0/24288073 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f2493fff700 1 --2- 172.21.15.77:0/24288073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249406c710 0x7f24940b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f2499095700 1 --2- 172.21.15.77:0/24288073 >> v2:172.21.15.159:3300/0 conn(0x7f24940af4d0 0x7f24940afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f24937fe700 1 -- 172.21.15.77:0/24288073 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958252715 0 0) 0x7f24940ae430 con 0x7f249415e080 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.181+0000 7f24937fe700 1 -- 172.21.15.77:0/24288073 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24840035b0 con 0x7f249415e080 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.182+0000 7f2493fff700 1 -- 172.21.15.77:0/24288073 >> v2:172.21.15.159:3300/0 conn(0x7f24940af4d0 msgr2=0x7f24940afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.182+0000 7f2493fff700 1 --2- 172.21.15.77:0/24288073 >> v2:172.21.15.159:3300/0 conn(0x7f24940af4d0 0x7f24940afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.182+0000 7f2493fff700 1 -- 172.21.15.77:0/24288073 >> v1:172.21.15.77:6789/0 conn(0x7f249415e080 legacy=0x7f249406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.182+0000 7f2493fff700 1 -- 172.21.15.77:0/24288073 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940ae710 con 0x7f249406c710 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.182+0000 7f2499095700 1 --2- 172.21.15.77:0/24288073 >> v2:172.21.15.159:3300/0 conn(0x7f24940af4d0 0x7f24940afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:01.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.182+0000 7f2493fff700 1 --2- 172.21.15.77:0/24288073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249406c710 0x7f24940b5050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2480009840 tx=0x7f2480009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba61bbe1e38fc2e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.185+0000 7f24937fe700 1 -- 172.21.15.77:0/24288073 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480011830 con 0x7f249406c710 2024-01-15T14:35:01.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f24937fe700 1 -- 172.21.15.77:0/24288073 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2480011990 con 0x7f249406c710 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f24937fe700 1 -- 172.21.15.77:0/24288073 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480011d20 con 0x7f249406c710 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 -- 172.21.15.77:0/24288073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249406c710 msgr2=0x7f24940b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 --2- 172.21.15.77:0/24288073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249406c710 0x7f24940b5050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2480009840 tx=0x7f2480009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 -- 172.21.15.77:0/24288073 shutdown_connections 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 --2- 172.21.15.77:0/24288073 >> v2:172.21.15.159:3300/0 conn(0x7f24940af4d0 0x7f24940afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 --2- 172.21.15.77:0/24288073 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249406c710 0x7f24940b5050 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 -- 172.21.15.77:0/24288073 >> 172.21.15.77:0/24288073 conn(0x7f2494071b90 msgr2=0x7f2494071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 -- 172.21.15.77:0/24288073 shutdown_connections 2024-01-15T14:35:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.186+0000 7f249aaf8700 1 -- 172.21.15.77:0/24288073 wait complete. 2024-01-15T14:35:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.187+0000 7f249aaf8700 1 Processor -- start 2024-01-15T14:35:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.187+0000 7f249aaf8700 1 -- start start 2024-01-15T14:35:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f249aaf8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 0x7f2494142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f249aaf8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24940af4d0 0x7f2494141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f249aaf8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249415e080 0x7f2494151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f249aaf8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2494156040 con 0x7f249406c710 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f249aaf8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24941450e0 con 0x7f24940af4d0 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f249aaf8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2494142660 con 0x7f249415e080 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f2498894700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 0x7f2494142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f2498894700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 0x7f2494142120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56080/0 (socket says 172.21.15.77:56080) 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.188+0000 7f2498894700 1 -- 172.21.15.77:0/2352640873 learned_addr learned my addr 172.21.15.77:0/2352640873 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2499095700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249415e080 0x7f2494151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2493fff700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24940af4d0 0x7f2494141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2498894700 1 -- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249415e080 msgr2=0x7f2494151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2498894700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249415e080 0x7f2494151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2498894700 1 -- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24940af4d0 msgr2=0x7f2494141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2498894700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24940af4d0 0x7f2494141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2498894700 1 -- 172.21.15.77:0/2352640873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c009a70 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2493fff700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24940af4d0 0x7f2494141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f2498894700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 0x7f2494142120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f248800c420 tx=0x7f248800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488005d70 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24880070d0 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488007440 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24800094a0 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.189+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2494155980 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.190+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24940753f0 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.190+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f24880075a0 con 0x7f249406c710 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.191+0000 7f24917fa700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f247806b350 0x7f247806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.191+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2488088b00 con 0x7f249406c710 2024-01-15T14:35:01.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.191+0000 7f2493fff700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f247806b350 0x7f247806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.191+0000 7f2493fff700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f247806b350 0x7f247806d810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2480009fd0 tx=0x7f2480008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.195+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2488053e00 con 0x7f249406c710 2024-01-15T14:35:01.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.390+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2494156180 con 0x7f249406c710 2024-01-15T14:35:01.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.391+0000 7f24917fa700 1 -- 172.21.15.77:0/2352640873 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f2488017020 con 0x7f249406c710 2024-01-15T14:35:01.392 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:01.393 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:34:55.161631+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:01.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f247806b350 msgr2=0x7f247806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f247806b350 0x7f247806d810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2480009fd0 tx=0x7f2480008030 comp rx=0 tx=0).stop 2024-01-15T14:35:01.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 msgr2=0x7f2494142120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 0x7f2494142120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f248800c420 tx=0x7f248800c7e0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 shutdown_connections 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f247806b350 0x7f247806d810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f249415e080 0x7f2494151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249406c710 0x7f2494142120 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 --2- 172.21.15.77:0/2352640873 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f24940af4d0 0x7f2494141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 >> 172.21.15.77:0/2352640873 conn(0x7f2494071b90 msgr2=0x7f249406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 shutdown_connections 2024-01-15T14:35:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.394+0000 7f249aaf8700 1 -- 172.21.15.77:0/2352640873 wait complete. 2024-01-15T14:35:01.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:01.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:01.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 Processor -- start 2024-01-15T14:35:01.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 -- start start 2024-01-15T14:35:01.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0b42a0 0x7f0abc06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0abc0ce470 0x7f0abc0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc06b090 con 0x7f0abc15dfb0 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc0cee40 con 0x7f0abc0ce470 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.531+0000 7f0ac3a70700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc06b300 con 0x7f0abc0b42a0 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac200d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0abc0ce470 0x7f0abc0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac180c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0abc15dfb0 0x7f0abc0d08f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48452/0 (socket says 172.21.15.77:48452) 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac180c700 1 -- 172.21.15.77:0/1059795882 learned_addr learned my addr 172.21.15.77:0/1059795882 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac100b700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0b42a0 0x7f0abc06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac200d700 1 -- 172.21.15.77:0/1059795882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0b42a0 msgr2=0x7f0abc06bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac200d700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0b42a0 0x7f0abc06bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac200d700 1 -- 172.21.15.77:0/1059795882 >> v1:172.21.15.77:6789/0 conn(0x7f0abc15dfb0 legacy=0x7f0abc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac080a700 1 -- 172.21.15.77:0/1059795882 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82828799 0 0) 0x7f0abc06b090 con 0x7f0abc15dfb0 2024-01-15T14:35:01.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac200d700 1 -- 172.21.15.77:0/1059795882 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc06b5e0 con 0x7f0abc0ce470 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac100b700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0b42a0 0x7f0abc06bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac200d700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc0ce470 0x7f0abc0ce900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009d90 tx=0x7f0aac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4185e03cf27310cb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac080a700 1 -- 172.21.15.77:0/1059795882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac00e070 con 0x7f0abc0ce470 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac080a700 1 -- 172.21.15.77:0/1059795882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac00bbd0 con 0x7f0abc0ce470 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.532+0000 7f0ac080a700 1 -- 172.21.15.77:0/1059795882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac012430 con 0x7f0abc0ce470 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1059795882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc0ce470 msgr2=0x7f0abc0ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc0ce470 0x7f0abc0ce900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009d90 tx=0x7f0aac00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1059795882 shutdown_connections 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc0ce470 0x7f0abc0ce900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1059795882 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0b42a0 0x7f0abc06bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1059795882 >> 172.21.15.77:0/1059795882 conn(0x7f0abc0b3780 msgr2=0x7f0abc0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:01.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1059795882 shutdown_connections 2024-01-15T14:35:01.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.533+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1059795882 wait complete. 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.534+0000 7f0ac3a70700 1 Processor -- start 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.534+0000 7f0ac3a70700 1 -- start start 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0abc0b42a0 0x7f0abc15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0ce470 0x7f0abc0c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 0x7f0abc06f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0c6110 con 0x7f0abc15dfb0 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0c63f0 con 0x7f0abc0b42a0 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0c66d0 con 0x7f0abc0ce470 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 0x7f0abc06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 0x7f0abc06f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56094/0 (socket says 172.21.15.77:56094) 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 -- 172.21.15.77:0/1267750136 learned_addr learned my addr 172.21.15.77:0/1267750136 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac100b700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0ce470 0x7f0abc0c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 -- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0ce470 msgr2=0x7f0abc0c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0ce470 0x7f0abc0c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac180c700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0abc0b42a0 0x7f0abc15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 -- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0abc0b42a0 msgr2=0x7f0abc15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0abc0b42a0 0x7f0abc15ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 -- 172.21.15.77:0/1267750136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac009a70 con 0x7f0abc15dfb0 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac200d700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 0x7f0abc06f730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0723f0 tx=0x7f0aac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0098f0 con 0x7f0abc15dfb0 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac020070 con 0x7f0abc15dfb0 2024-01-15T14:35:01.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc0c6eb0 con 0x7f0abc15dfb0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac004260 con 0x7f0abc15dfb0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.535+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc15d2d0 con 0x7f0abc15dfb0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.536+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc069020 con 0x7f0abc15dfb0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.537+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0aac0043c0 con 0x7f0abc15dfb0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.537+0000 7f0aba7fc700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0aa806b350 0x7f0aa806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.537+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0aac022070 con 0x7f0abc15dfb0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.537+0000 7f0ac180c700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0aa806b350 0x7f0aa806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.538+0000 7f0ac180c700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0aa806b350 0x7f0aa806d810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4000f60 tx=0x7f0ab4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.543+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0aac056ef0 con 0x7f0abc15dfb0 2024-01-15T14:35:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.732+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0abc0b7ab0 con 0x7f0abc15dfb0 2024-01-15T14:35:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.732+0000 7f0aba7fc700 1 -- 172.21.15.77:0/1267750136 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0aac05a510 con 0x7f0abc15dfb0 2024-01-15T14:35:01.733 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:01.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0aa806b350 msgr2=0x7f0aa806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0aa806b350 0x7f0aa806d810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4000f60 tx=0x7f0ab4009040 comp rx=0 tx=0).stop 2024-01-15T14:35:01.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 msgr2=0x7f0abc06f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 0x7f0abc06f730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0723f0 tx=0x7f0aac004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 shutdown_connections 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0aa806b350 0x7f0aa806d810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0abc15dfb0 0x7f0abc06f730 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0abc0b42a0 0x7f0abc15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 --2- 172.21.15.77:0/1267750136 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0abc0ce470 0x7f0abc0c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 >> 172.21.15.77:0/1267750136 conn(0x7f0abc0b3780 msgr2=0x7f0abc0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 shutdown_connections 2024-01-15T14:35:01.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.734+0000 7f0ac3a70700 1 -- 172.21.15.77:0/1267750136 wait complete. 2024-01-15T14:35:01.748 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T14:35:01.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.867+0000 7fa382ff2700 1 Processor -- start 2024-01-15T14:35:01.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.868+0000 7fa382ff2700 1 -- start start 2024-01-15T14:35:01.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.868+0000 7fa382ff2700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa37c1446a0 0x7fa37c0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.868+0000 7fa382ff2700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa37c06c710 0x7fa37c0b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.868+0000 7fa382ff2700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0af950 con 0x7fa37c0adf00 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.868+0000 7fa382ff2700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c144a80 con 0x7fa37c06c710 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.868+0000 7fa382ff2700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c0afc30 con 0x7fa37c1446a0 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa37c06c710 0x7fa37c0b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa37c06c710 0x7fa37c0b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56110/0 (socket says 172.21.15.77:56110) 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 -- 172.21.15.77:0/3534305338 learned_addr learned my addr 172.21.15.77:0/3534305338 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa380d8e700 1 --2- 172.21.15.77:0/3534305338 >> v2:172.21.15.159:3300/0 conn(0x7fa37c1446a0 0x7fa37c0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 -- 172.21.15.77:0/3534305338 >> v2:172.21.15.159:3300/0 conn(0x7fa37c1446a0 msgr2=0x7fa37c0b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 --2- 172.21.15.77:0/3534305338 >> v2:172.21.15.159:3300/0 conn(0x7fa37c1446a0 0x7fa37c0b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 -- 172.21.15.77:0/3534305338 >> v1:172.21.15.159:6789/0 conn(0x7fa37c0adf00 legacy=0x7fa37c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 -- 172.21.15.77:0/3534305338 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0aff10 con 0x7fa37c06c710 2024-01-15T14:35:01.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa38158f700 1 --2- 172.21.15.77:0/3534305338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c06c710 0x7fa37c0b3d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0729a0 tx=0x7fa37400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40dc76039051b57 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa37b7fe700 1 -- 172.21.15.77:0/3534305338 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400e070 con 0x7fa37c06c710 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa37b7fe700 1 -- 172.21.15.77:0/3534305338 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37400b9c0 con 0x7fa37c06c710 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.869+0000 7fa37b7fe700 1 -- 172.21.15.77:0/3534305338 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400bd70 con 0x7fa37c06c710 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 -- 172.21.15.77:0/3534305338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c06c710 msgr2=0x7fa37c0b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 --2- 172.21.15.77:0/3534305338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c06c710 0x7fa37c0b3d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0729a0 tx=0x7fa37400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 -- 172.21.15.77:0/3534305338 shutdown_connections 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 --2- 172.21.15.77:0/3534305338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c06c710 0x7fa37c0b3d40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 --2- 172.21.15.77:0/3534305338 >> v2:172.21.15.159:3300/0 conn(0x7fa37c1446a0 0x7fa37c0b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 -- 172.21.15.77:0/3534305338 >> 172.21.15.77:0/3534305338 conn(0x7fa37c071b90 msgr2=0x7fa37c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 -- 172.21.15.77:0/3534305338 shutdown_connections 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.870+0000 7fa382ff2700 1 -- 172.21.15.77:0/3534305338 wait complete. 2024-01-15T14:35:01.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.871+0000 7fa382ff2700 1 Processor -- start 2024-01-15T14:35:01.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.871+0000 7fa382ff2700 1 -- start start 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa382ff2700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa37c06c710 0x7fa37c15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa382ff2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 0x7fa37c15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa382ff2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa37c1446a0 0x7fa37c13c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa382ff2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c13c720 con 0x7fa37c0adf00 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa382ff2700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c13c950 con 0x7fa37c06c710 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa382ff2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c13cc30 con 0x7fa37c1446a0 2024-01-15T14:35:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.872+0000 7fa37bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 0x7fa37c15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 0x7fa37c15b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56122/0 (socket says 172.21.15.77:56122) 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 -- 172.21.15.77:0/2933201868 learned_addr learned my addr 172.21.15.77:0/2933201868 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa380d8e700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa37c06c710 0x7fa37c15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa38158f700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa37c1446a0 0x7fa37c13c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 -- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa37c1446a0 msgr2=0x7fa37c13c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa37c1446a0 0x7fa37c13c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 -- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa37c06c710 msgr2=0x7fa37c15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa37c06c710 0x7fa37c15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 -- 172.21.15.77:0/2933201868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa374009b30 con 0x7fa37c0adf00 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa37bfff700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 0x7fa37c15b670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa36800c450 tx=0x7fa36800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680052d0 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa368005430 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680038c0 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c13cf10 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.873+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c13d1f0 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.874+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c0753f0 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.874+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa368003a20 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.875+0000 7fa3797fa700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa36006b350 0x7fa36006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.875+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa368088aa0 con 0x7fa37c0adf00 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.875+0000 7fa380d8e700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa36006b350 0x7fa36006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.875+0000 7fa380d8e700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa36006b350 0x7fa36006d810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa370007a30 tx=0x7fa370008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:01.880+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa368053cf0 con 0x7fa37c0adf00 2024-01-15T14:35:02.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.070+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa37c15d020 con 0x7fa37c0adf00 2024-01-15T14:35:02.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.070+0000 7fa3797fa700 1 -- 172.21.15.77:0/2933201868 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa37c15d020 con 0x7fa37c0adf00 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:02.072 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa36006b350 msgr2=0x7fa36006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa36006b350 0x7fa36006d810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa370007a30 tx=0x7fa370008040 comp rx=0 tx=0).stop 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 msgr2=0x7fa37c15b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 0x7fa37c15b670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa36800c450 tx=0x7fa36800c760 comp rx=0 tx=0).stop 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 shutdown_connections 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa36006b350 0x7fa36006d810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa37c1446a0 0x7fa37c13c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa37c06c710 0x7fa37c15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 --2- 172.21.15.77:0/2933201868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa37c0adf00 0x7fa37c15b670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 >> 172.21.15.77:0/2933201868 conn(0x7fa37c071b90 msgr2=0x7fa37c06bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:02.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 shutdown_connections 2024-01-15T14:35:02.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.072+0000 7fa382ff2700 1 -- 172.21.15.77:0/2933201868 wait complete. 2024-01-15T14:35:02.086 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T14:35:02.086 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T14:35:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45f1443700 1 Processor -- start 2024-01-15T14:35:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45f1443700 1 -- start start 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45f1443700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45ec15e3c0 0x7f45ec0b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45f1443700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ec0b6280 con 0x7f45ec15dfe0 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45f1443700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ec0d1e60 con 0x7f45ec0b5e70 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45f1443700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ec0d2ae0 con 0x7f45ec15e3c0 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45ea7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45ec15e3c0 0x7f45ec0b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45eaffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f45ec15dfe0 0x7f45ec0d3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48464/0 (socket says 172.21.15.77:48464) 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45eaffd700 1 -- 172.21.15.77:0/2231994905 learned_addr learned my addr 172.21.15.77:0/2231994905 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2231994905 >> v1:172.21.15.159:6789/0 conn(0x7f45ec0b5e70 legacy=0x7f45ec0b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2231994905 >> v1:172.21.15.77:6789/0 conn(0x7f45ec15dfe0 legacy=0x7f45ec0d3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2231994905 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec0d2140 con 0x7f45ec15e3c0 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45ea7fc700 1 --2- 172.21.15.77:0/2231994905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15e3c0 0x7f45ec0b5930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009840 tx=0x7f45e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5072736352229e61 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45e9ffb700 1 -- 172.21.15.77:0/2231994905 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0011830 con 0x7f45ec15e3c0 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45e9ffb700 1 -- 172.21.15.77:0/2231994905 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e0011990 con 0x7f45ec15e3c0 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.212+0000 7f45e9ffb700 1 -- 172.21.15.77:0/2231994905 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0011d20 con 0x7f45ec15e3c0 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 -- 172.21.15.77:0/2231994905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15e3c0 msgr2=0x7f45ec0b5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 --2- 172.21.15.77:0/2231994905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15e3c0 0x7f45ec0b5930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009840 tx=0x7f45e0009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 -- 172.21.15.77:0/2231994905 shutdown_connections 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 --2- 172.21.15.77:0/2231994905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15e3c0 0x7f45ec0b5930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 -- 172.21.15.77:0/2231994905 >> 172.21.15.77:0/2231994905 conn(0x7f45ec0d0b10 msgr2=0x7f45ec0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:02.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 -- 172.21.15.77:0/2231994905 shutdown_connections 2024-01-15T14:35:02.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.213+0000 7f45f1443700 1 -- 172.21.15.77:0/2231994905 wait complete. 2024-01-15T14:35:02.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.214+0000 7f45f1443700 1 Processor -- start 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.214+0000 7f45f1443700 1 -- start start 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45f1443700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ec0b5e70 0x7f45ec0ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45f1443700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 0x7f45ec1f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45f1443700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ec15e3c0 0x7f45ec1f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45f1443700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0c8880 con 0x7f45ec15dfe0 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45f1443700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0c8b60 con 0x7f45ec15e3c0 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45f1443700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0c8e40 con 0x7f45ec0b5e70 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 0x7f45ec1f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 0x7f45ec1f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56140/0 (socket says 172.21.15.77:56140) 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2808428208 learned_addr learned my addr 172.21.15.77:0/2808428208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45eb7fe700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ec15e3c0 0x7f45ec1f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45eaffd700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ec0b5e70 0x7f45ec0ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ec0b5e70 msgr2=0x7f45ec0ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ec0b5e70 0x7f45ec0ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ec15e3c0 msgr2=0x7f45ec1f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ec15e3c0 0x7f45ec1f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.215+0000 7f45ea7fc700 1 -- 172.21.15.77:0/2808428208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e00094a0 con 0x7f45ec15dfe0 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.216+0000 7f45ea7fc700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 0x7f45ec1f6c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.216+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e00036a0 con 0x7f45ec15dfe0 2024-01-15T14:35:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.216+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e001e070 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.216+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec0c9120 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.216+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0003a40 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.216+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec0ca5b0 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.217+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ec0ca740 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.217+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f45e0003ba0 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.218+0000 7f45d37fe700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f45d806b350 0x7f45d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.218+0000 7f45eaffd700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f45d806b350 0x7f45d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.218+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f45e008a7d0 con 0x7f45ec15dfe0 2024-01-15T14:35:02.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.218+0000 7f45eaffd700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f45d806b350 0x7f45d806d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f45dc005910 tx=0x7f45dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:02.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.223+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45e0055a20 con 0x7f45ec15dfe0 2024-01-15T14:35:02.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:02.415+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f45ec0d3b40 con 0x7f45ec15dfe0 2024-01-15T14:35:03.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:35:03.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:35:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.186+0000 7f45d37fe700 1 -- 172.21.15.77:0/2808428208 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f45e00200b0 con 0x7f45ec15dfe0 2024-01-15T14:35:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f45d806b350 msgr2=0x7f45d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f45d806b350 0x7f45d806d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f45dc005910 tx=0x7f45dc009040 comp rx=0 tx=0).stop 2024-01-15T14:35:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 msgr2=0x7f45ec1f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 0x7f45ec1f6c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0004750 comp rx=0 tx=0).stop 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 shutdown_connections 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f45d806b350 0x7f45d806d810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f45ec0b5e70 0x7f45ec0ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f45ec15e3c0 0x7f45ec1f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 --2- 172.21.15.77:0/2808428208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45ec15dfe0 0x7f45ec1f6c10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 >> 172.21.15.77:0/2808428208 conn(0x7f45ec0d0b10 msgr2=0x7f45ec0b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 shutdown_connections 2024-01-15T14:35:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.189+0000 7f45f1443700 1 -- 172.21.15.77:0/2808428208 wait complete. 2024-01-15T14:35:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-01-15T14:35:03.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T14:35:03.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.334+0000 7f38e9244700 1 Processor -- start 2024-01-15T14:35:03.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.334+0000 7f38e9244700 1 -- start start 2024-01-15T14:35:03.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.334+0000 7f38e9244700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e415bcd0 0x7f38e40cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:03.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e9244700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38e40713d0 0x7f38e4071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:03.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e9244700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e415c7c0 con 0x7f38e40ce3a0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e9244700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e4071dc0 con 0x7f38e40713d0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e9244700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e406d4d0 con 0x7f38e415bcd0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38e40713d0 0x7f38e4071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38e40713d0 0x7f38e4071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56144/0 (socket says 172.21.15.77:56144) 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 -- 172.21.15.77:0/3470467348 learned_addr learned my addr 172.21.15.77:0/3470467348 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e2d9d700 1 --2- 172.21.15.77:0/3470467348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e415bcd0 0x7f38e40cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 -- 172.21.15.77:0/3470467348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e415bcd0 msgr2=0x7f38e40cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 --2- 172.21.15.77:0/3470467348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e415bcd0 0x7f38e40cfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 -- 172.21.15.77:0/3470467348 >> v1:172.21.15.159:6789/0 conn(0x7f38e40ce3a0 legacy=0x7f38e415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 -- 172.21.15.77:0/3470467348 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e406d7b0 con 0x7f38e40713d0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e359e700 1 --2- 172.21.15.77:0/3470467348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4071880 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009e50 tx=0x7f38dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380c969924b4d1aa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e1d9b700 1 -- 172.21.15.77:0/3470467348 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc00e070 con 0x7f38e40713d0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e1d9b700 1 -- 172.21.15.77:0/3470467348 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc00b9c0 con 0x7f38e40713d0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.335+0000 7f38e1d9b700 1 -- 172.21.15.77:0/3470467348 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc00bd70 con 0x7f38e40713d0 2024-01-15T14:35:03.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 -- 172.21.15.77:0/3470467348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 msgr2=0x7f38e4071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 --2- 172.21.15.77:0/3470467348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4071880 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009e50 tx=0x7f38dc00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 -- 172.21.15.77:0/3470467348 shutdown_connections 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 --2- 172.21.15.77:0/3470467348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4071880 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 --2- 172.21.15.77:0/3470467348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e415bcd0 0x7f38e40cfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 -- 172.21.15.77:0/3470467348 >> 172.21.15.77:0/3470467348 conn(0x7f38e406aff0 msgr2=0x7f38e406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 -- 172.21.15.77:0/3470467348 shutdown_connections 2024-01-15T14:35:03.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.336+0000 7f38e9244700 1 -- 172.21.15.77:0/3470467348 wait complete. 2024-01-15T14:35:03.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.337+0000 7f38e9244700 1 Processor -- start 2024-01-15T14:35:03.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.337+0000 7f38e9244700 1 -- start start 2024-01-15T14:35:03.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e9244700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e9244700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e40ce3a0 0x7f38e41537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e9244700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38e415bcd0 0x7f38e40a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e9244700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e41462e0 con 0x7f38e40713d0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e9244700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e4153cf0 con 0x7f38e415bcd0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e9244700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40ac620 con 0x7f38e40ce3a0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56150/0 (socket says 172.21.15.77:56150) 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 -- 172.21.15.77:0/3453239455 learned_addr learned my addr 172.21.15.77:0/3453239455 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e259c700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e40ce3a0 0x7f38e41537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e359e700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38e415bcd0 0x7f38e40a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 -- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e40ce3a0 msgr2=0x7f38e41537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e40ce3a0 0x7f38e41537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 -- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38e415bcd0 msgr2=0x7f38e40a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.338+0000 7f38e2d9d700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38e415bcd0 0x7f38e40a9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38e2d9d700 1 -- 172.21.15.77:0/3453239455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc009b30 con 0x7f38e40713d0 2024-01-15T14:35:03.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38e359e700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38e415bcd0 0x7f38e40a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38e2d9d700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4145da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38d400d800 tx=0x7f38d400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d40125f0 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d4012750 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e40ac900 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4012b00 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.339+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e40acec0 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.340+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e40753f0 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.340+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f38d4012c60 con 0x7f38e40713d0 2024-01-15T14:35:03.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.341+0000 7f38cb7fe700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f38d006b350 0x7f38d006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:03.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.341+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f38d408a770 con 0x7f38e40713d0 2024-01-15T14:35:03.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.341+0000 7f38e259c700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f38d006b350 0x7f38d006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:03.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.341+0000 7f38e259c700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f38d006b350 0x7f38d006d810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38e415f080 tx=0x7f38d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:03.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.345+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38d4055830 con 0x7f38e40713d0 2024-01-15T14:35:03.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:03.536+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f38e40cef10 con 0x7f38e40713d0 2024-01-15T14:35:04.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.189+0000 7f38cb7fe700 1 -- 172.21.15.77:0/3453239455 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f38d4058e50 con 0x7f38e40713d0 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f38d006b350 msgr2=0x7f38d006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f38d006b350 0x7f38d006d810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38e415f080 tx=0x7f38d8009040 comp rx=0 tx=0).stop 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 msgr2=0x7f38e4145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4145da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38d400d800 tx=0x7f38d400db10 comp rx=0 tx=0).stop 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 shutdown_connections 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f38d006b350 0x7f38d006d810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f38e415bcd0 0x7f38e40a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38e40713d0 0x7f38e4145da0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 --2- 172.21.15.77:0/3453239455 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38e40ce3a0 0x7f38e41537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 >> 172.21.15.77:0/3453239455 conn(0x7f38e406aff0 msgr2=0x7f38e40af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:04.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 shutdown_connections 2024-01-15T14:35:04.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.192+0000 7f38e9244700 1 -- 172.21.15.77:0/3453239455 wait complete. 2024-01-15T14:35:04.193 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-01-15T14:35:04.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T14:35:04.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.342+0000 7f9cc48f2700 1 Processor -- start 2024-01-15T14:35:04.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.342+0000 7f9cc48f2700 1 -- start start 2024-01-15T14:35:04.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cc48f2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9cc00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cc48f2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc00d0a70 con 0x7f9cc0161690 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cc48f2700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc00d1c10 con 0x7f9cc0161a70 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cc48f2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00b2ab0 con 0x7f9cc009b110 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbe59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9cc0161690 0x7f9cc009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48476/0 (socket says 172.21.15.77:48476) 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbe59c700 1 -- 172.21.15.77:0/3235328961 learned_addr learned my addr 172.21.15.77:0/3235328961 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735903186 0 0) 0x7f9cc00d0a70 con 0x7f9cc0161690 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca80035b0 con 0x7f9cc0161690 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbed9d700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9cc00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401801831 0 0) 0x7f9ca80035b0 con 0x7f9cc0161690 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 msgr2=0x7f9cc00d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9cc00d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 >> v1:172.21.15.159:6789/0 conn(0x7f9cc0161a70 legacy=0x7f9cc00cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc00b2d90 con 0x7f9cc0161690 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f9cb0001670 con 0x7f9cc0161690 2024-01-15T14:35:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbed9d700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9cc00d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.343+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 >> v1:172.21.15.77:6789/0 conn(0x7f9cc0161690 legacy=0x7f9cc009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ca8003a30 0x7f9ca8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9ca8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9ca8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00b2ab0 con 0x7f9ca8004850 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca800bb40 con 0x7f9ca8003a30 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca800bd00 con 0x7f9cc009b110 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9ca8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbe59c700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ca8003a30 0x7f9ca8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbed9d700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9ca8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 -- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 msgr2=0x7f9ca8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc009b110 0x7f9ca8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 -- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ca8003a30 msgr2=0x7f9ca8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9ca8003a30 0x7f9ca8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 -- 172.21.15.77:0/3235328961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80035b0 con 0x7f9ca8004850 2024-01-15T14:35:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9ca8004cd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40017c0 tx=0x7f9ca8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73772cf75c7a789 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40052d0 con 0x7f9ca8004850 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb4005430 con 0x7f9ca8004850 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.344+0000 7f9cbd59a700 1 -- 172.21.15.77:0/3235328961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40038c0 con 0x7f9ca8004850 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cc48f2700 1 -- 172.21.15.77:0/3235328961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 msgr2=0x7f9ca8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/3235328961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9ca8004cd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40017c0 tx=0x7f9ca8005880 comp rx=0 tx=0).stop 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cbed9d700 1 -- 172.21.15.77:0/3235328961 reap_dead start 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cc48f2700 1 -- 172.21.15.77:0/3235328961 shutdown_connections 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cc48f2700 1 -- 172.21.15.77:0/3235328961 >> 172.21.15.77:0/3235328961 conn(0x7f9cc00d1e50 msgr2=0x7f9cc00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cc48f2700 1 -- 172.21.15.77:0/3235328961 shutdown_connections 2024-01-15T14:35:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.345+0000 7f9cc48f2700 1 -- 172.21.15.77:0/3235328961 wait complete. 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.346+0000 7f9cc48f2700 1 Processor -- start 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.346+0000 7f9cc48f2700 1 -- start start 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9cc0152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc00c8400 0x7f9cc00c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cc00c9460 0x7f9cc00c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00c7360 con 0x7f9ca8004850 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00c7640 con 0x7f9cc00c9460 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00c7920 con 0x7f9cc00c8400 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9cc0152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9cc0152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56166/0 (socket says 172.21.15.77:56166) 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 -- 172.21.15.77:0/4162863807 learned_addr learned my addr 172.21.15.77:0/4162863807 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc00c8400 0x7f9cc00c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbed9d700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cc00c9460 0x7f9cc00c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 -- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc00c8400 msgr2=0x7f9cc00c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc00c8400 0x7f9cc00c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 -- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cc00c9460 msgr2=0x7f9cc00c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cc00c9460 0x7f9cc00c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 -- 172.21.15.77:0/4162863807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc00c7c00 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbed9d700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cc00c9460 0x7f9cc00c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cbe59c700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9cc0152c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0007290 tx=0x7f9cb00075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb00014c0 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0007780 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc00c7ee0 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.347+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000fb10 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.348+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc00c6f30 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.348+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc00c7120 con 0x7f9ca8004850 2024-01-15T14:35:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.349+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9cb000fc70 con 0x7f9ca8004850 2024-01-15T14:35:04.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.349+0000 7f9caeffd700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9ca006b350 0x7f9ca006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:04.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.349+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9cb0089c10 con 0x7f9ca8004850 2024-01-15T14:35:04.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.349+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9ca006b350 0x7f9ca006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:04.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.350+0000 7f9cbdd9b700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9ca006b350 0x7f9ca006d810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000ff0 tx=0x7f9cb4009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:04.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.354+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cc00c7120 con 0x7f9ca8004850 2024-01-15T14:35:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:04.588+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9cc00c7120 con 0x7f9ca8004850 2024-01-15T14:35:05.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.213+0000 7f9caeffd700 1 -- 172.21.15.77:0/4162863807 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9cc00c7120 con 0x7f9ca8004850 2024-01-15T14:35:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.218+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9ca006b350 msgr2=0x7f9ca006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.218+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9ca006b350 0x7f9ca006d810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000ff0 tx=0x7f9cb4009030 comp rx=0 tx=0).stop 2024-01-15T14:35:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.218+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 msgr2=0x7f9cc0152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.218+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9cc0152c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0007290 tx=0x7f9cb00075a0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.219+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 shutdown_connections 2024-01-15T14:35:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.219+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9ca006b350 0x7f9ca006d810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.219+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9cc00c9460 0x7f9cc00c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.219+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ca8004850 0x7f9cc0152c80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.219+0000 7f9cc48f2700 1 --2- 172.21.15.77:0/4162863807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9cc00c8400 0x7f9cc00c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.219+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 >> 172.21.15.77:0/4162863807 conn(0x7f9cc00d1e50 msgr2=0x7f9cc015e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.220+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 shutdown_connections 2024-01-15T14:35:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.220+0000 7f9cc48f2700 1 -- 172.21.15.77:0/4162863807 wait complete. 2024-01-15T14:35:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 2 and data pool 3 2024-01-15T14:35:05.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:05.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.371+0000 7f19c3f1d700 1 Processor -- start 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.371+0000 7f19c3f1d700 1 -- start start 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c3f1d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f19bc15e080 0x7f19bc0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c3f1d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19bc0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c3f1d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc0d0150 con 0x7f19bc0ce4d0 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c3f1d700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19bc15e460 con 0x7f19bc15e080 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c3f1d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0cee50 con 0x7f19bc0d22f0 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c14b8700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f19bc0ce4d0 0x7f19bc0d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48486/0 (socket says 172.21.15.77:48486) 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c14b8700 1 -- 172.21.15.77:0/3711020968 learned_addr learned my addr 172.21.15.77:0/3711020968 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561089585 0 0) 0x7f19bc0d0150 con 0x7f19bc0ce4d0 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c24ba700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19bc0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/3711020968 >> v2:172.21.15.159:3300/0 conn(0x7f19bc15e080 0x7f19bc0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a80035b0 con 0x7f19bc0ce4d0 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554129240 0 0) 0x7f19a80035b0 con 0x7f19bc0ce4d0 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 msgr2=0x7f19bc0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19bc0d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 >> v2:172.21.15.159:3300/0 conn(0x7f19bc15e080 msgr2=0x7f19bc0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 --2- 172.21.15.77:0/3711020968 >> v2:172.21.15.159:3300/0 conn(0x7f19bc15e080 0x7f19bc0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc0cf130 con 0x7f19bc0ce4d0 2024-01-15T14:35:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f19ac001670 con 0x7f19bc0ce4d0 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 >> v1:172.21.15.77:6789/0 conn(0x7f19bc0ce4d0 legacy=0x7f19bc0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19a8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc15e460 con 0x7f19a8003a00 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0cee50 con 0x7f19a8004d90 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.372+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a800ba10 con 0x7f19bc0d22f0 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19a8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c14b8700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c24ba700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 -- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 msgr2=0x7f19a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 -- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19a8004d90 msgr2=0x7f19a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 -- 172.21.15.77:0/3711020968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a80035b0 con 0x7f19a8003a00 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c14b8700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc0d22f0 0x7f19a8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c24ba700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19a8004d90 0x7f19a8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19a8003e20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009b90 tx=0x7f19a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a9a9fbdd07c4e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800e6c0 con 0x7f19a8003a00 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b800e820 con 0x7f19a8003a00 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c0cb7700 1 -- 172.21.15.77:0/3711020968 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800eb90 con 0x7f19a8003a00 2024-01-15T14:35:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c3f1d700 1 -- 172.21.15.77:0/3711020968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 msgr2=0x7f19a8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.373+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/3711020968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19a8003e20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009b90 tx=0x7f19a8005880 comp rx=0 tx=0).stop 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.374+0000 7f19c24ba700 1 -- 172.21.15.77:0/3711020968 reap_dead start 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.374+0000 7f19c3f1d700 1 -- 172.21.15.77:0/3711020968 shutdown_connections 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.374+0000 7f19c3f1d700 1 -- 172.21.15.77:0/3711020968 >> 172.21.15.77:0/3711020968 conn(0x7f19bc073130 msgr2=0x7f19bc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.374+0000 7f19c3f1d700 1 -- 172.21.15.77:0/3711020968 shutdown_connections 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.374+0000 7f19c3f1d700 1 -- 172.21.15.77:0/3711020968 wait complete. 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.375+0000 7f19c3f1d700 1 Processor -- start 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.375+0000 7f19c3f1d700 1 -- start start 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.375+0000 7f19c3f1d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19bc0c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.375+0000 7f19c3f1d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19bc0c6140 0x7f19bc0c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c3f1d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc15c400 0x7f19bc15c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c3f1d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc1e6780 con 0x7f19a8003a00 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c3f1d700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c91a0 con 0x7f19bc0c6140 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c3f1d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c71f0 con 0x7f19bc15c400 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19bc0c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19bc0c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56180/0 (socket says 172.21.15.77:56180) 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 -- 172.21.15.77:0/4003090465 learned_addr learned my addr 172.21.15.77:0/4003090465 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c14b8700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19bc0c6140 0x7f19bc0c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c24ba700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc15c400 0x7f19bc15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 -- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc15c400 msgr2=0x7f19bc15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc15c400 0x7f19bc15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 -- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19bc0c6140 msgr2=0x7f19bc0c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19bc0c6140 0x7f19bc0c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 -- 172.21.15.77:0/4003090465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc0c6c30 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c1cb9700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19bc0c7ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19b800e4b0 tx=0x7f19b80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8021070 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b80045c0 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc0c6f10 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800e6c0 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.376+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc15cdb0 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.377+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc067f60 con 0x7f19a8003a00 2024-01-15T14:35:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.377+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f19b8025070 con 0x7f19a8003a00 2024-01-15T14:35:05.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.378+0000 7f19b27fc700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f19a006b350 0x7f19a006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.378+0000 7f19c14b8700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f19a006b350 0x7f19a006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.378+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19b808ac60 con 0x7f19a8003a00 2024-01-15T14:35:05.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.378+0000 7f19c14b8700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f19a006b350 0x7f19a006d810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19ac000fb0 tx=0x7f19ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:05.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.383+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19b8055b60 con 0x7f19a8003a00 2024-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.576+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19bc0d19c0 con 0x7f19a8003a00 2024-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.577+0000 7f19b27fc700 1 -- 172.21.15.77:0/4003090465 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f19b802c090 con 0x7f19a8003a00 2024-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:04.589568+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f19a006b350 msgr2=0x7f19a006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f19a006b350 0x7f19a006d810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19ac000fb0 tx=0x7f19ac009040 comp rx=0 tx=0).stop 2024-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 msgr2=0x7f19bc0c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19bc0c7ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19b800e4b0 tx=0x7f19b80047c0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 shutdown_connections 2024-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f19a006b350 0x7f19a006d810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f19bc15c400 0x7f19bc15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a8003a00 0x7f19bc0c7ac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 --2- 172.21.15.77:0/4003090465 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f19bc0c6140 0x7f19bc0c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 >> 172.21.15.77:0/4003090465 conn(0x7f19bc073130 msgr2=0x7f19bc0b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 shutdown_connections 2024-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.579+0000 7f19c3f1d700 1 -- 172.21.15.77:0/4003090465 wait complete. 2024-01-15T14:35:05.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.713+0000 7f094b170700 1 Processor -- start 2024-01-15T14:35:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.713+0000 7f094b170700 1 -- start start 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094b170700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 0x7f09440cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094b170700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09440b49a0 0x7f09440d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094b170700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440d09a0 con 0x7f0944152cc0 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094b170700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094415c4b0 con 0x7f09440b49a0 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094b170700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094415c790 con 0x7f094406c710 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f0948f0c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0944152cc0 0x7f094406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48496/0 (socket says 172.21.15.77:48496) 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f0948f0c700 1 -- 172.21.15.77:0/3844464871 learned_addr learned my addr 172.21.15.77:0/3844464871 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f0943fff700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 0x7f09440cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094970d700 1 --2- 172.21.15.77:0/3844464871 >> v2:172.21.15.159:3300/0 conn(0x7f09440b49a0 0x7f09440d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068312916 0 0) 0x7f09440d09a0 con 0x7f0944152cc0 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09340035b0 con 0x7f0944152cc0 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2877904084 0 0) 0x7f09340035b0 con 0x7f0944152cc0 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 msgr2=0x7f09440cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 0x7f09440cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 >> v2:172.21.15.159:3300/0 conn(0x7f09440b49a0 msgr2=0x7f09440d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 --2- 172.21.15.77:0/3844464871 >> v2:172.21.15.159:3300/0 conn(0x7f09440b49a0 0x7f09440d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094415ca70 con 0x7f0944152cc0 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f0938001670 con 0x7f0944152cc0 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 >> v1:172.21.15.77:6789/0 conn(0x7f0944152cc0 legacy=0x7f094406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 0x7f0934003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0934003f80 0x7f0934004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f0934005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094415c4b0 con 0x7f0934004d90 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094415c790 con 0x7f0934003f80 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093400ba60 con 0x7f094406c710 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.714+0000 7f094970d700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f0934005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f0943fff700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0934003f80 0x7f0934004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f0948f0c700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 0x7f0934003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 -- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 msgr2=0x7f0934003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094406c710 0x7f0934003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 -- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0934003f80 msgr2=0x7f0934004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0934003f80 0x7f0934004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 -- 172.21.15.77:0/3844464871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340035b0 con 0x7f0934004d90 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f0934005210 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e170 tx=0x7f0934005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eee76baf7b16b07 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c00ec90 con 0x7f0934004d90 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00f040 con 0x7f0934004d90 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f09437fe700 1 -- 172.21.15.77:0/3844464871 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c014480 con 0x7f0934004d90 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094b170700 1 -- 172.21.15.77:0/3844464871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 msgr2=0x7f0934005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094b170700 1 --2- 172.21.15.77:0/3844464871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f0934005210 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e170 tx=0x7f0934005880 comp rx=0 tx=0).stop 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094970d700 1 -- 172.21.15.77:0/3844464871 reap_dead start 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.715+0000 7f094b170700 1 -- 172.21.15.77:0/3844464871 shutdown_connections 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.716+0000 7f094b170700 1 -- 172.21.15.77:0/3844464871 >> 172.21.15.77:0/3844464871 conn(0x7f0944071b90 msgr2=0x7f0944071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.716+0000 7f094b170700 1 -- 172.21.15.77:0/3844464871 shutdown_connections 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.716+0000 7f094b170700 1 -- 172.21.15.77:0/3844464871 wait complete. 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.717+0000 7f094b170700 1 Processor -- start 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.717+0000 7f094b170700 1 -- start start 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f094414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094414dec0 0x7f09441403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09441f1cb0 0x7f0944151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094414b670 con 0x7f0934004d90 2024-01-15T14:35:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094414aa10 con 0x7f09441f1cb0 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0944154a80 con 0x7f094414dec0 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f094414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f094414b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56204/0 (socket says 172.21.15.77:56204) 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 -- 172.21.15.77:0/2641844262 learned_addr learned my addr 172.21.15.77:0/2641844262 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 -- 172.21.15.77:0/2641844262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094414dec0 msgr2=0x7f09441403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094414dec0 0x7f09441403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 -- 172.21.15.77:0/2641844262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09441f1cb0 msgr2=0x7f0944151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09441f1cb0 0x7f0944151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 -- 172.21.15.77:0/2641844262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944154d60 con 0x7f0934004d90 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f0948f0c700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f094414b020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0938001a30 tx=0x7f09380076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:05.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093800f820 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093800f980 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0944155040 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093800fd30 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.718+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094414e340 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.719+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09440753f0 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.720+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f093801d460 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.720+0000 7f09417fa700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f092806b350 0x7f092806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.720+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0938089cf0 con 0x7f0934004d90 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.720+0000 7f0943fff700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f092806b350 0x7f092806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:05.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.720+0000 7f0943fff700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f092806b350 0x7f092806d810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0944152670 tx=0x7f09300093c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:05.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.725+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f093808e070 con 0x7f0934004d90 2024-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.916+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0944067f60 con 0x7f0934004d90 2024-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.917+0000 7f09417fa700 1 -- 172.21.15.77:0/2641844262 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f09380585f0 con 0x7f0934004d90 2024-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:04.589568+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f092806b350 msgr2=0x7f092806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f092806b350 0x7f092806d810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0944152670 tx=0x7f09300093c0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 msgr2=0x7f094414b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f094414b020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0938001a30 tx=0x7f09380076a0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 shutdown_connections 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f092806b350 0x7f092806d810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09441f1cb0 0x7f0944151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0934004d90 0x7f094414b020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 --2- 172.21.15.77:0/2641844262 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f094414dec0 0x7f09441403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 >> 172.21.15.77:0/2641844262 conn(0x7f0944071b90 msgr2=0x7f09440b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 shutdown_connections 2024-01-15T14:35:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:05.920+0000 7f094b170700 1 -- 172.21.15.77:0/2641844262 wait complete. 2024-01-15T14:35:05.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T14:35:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.057+0000 7fe3b20d0700 1 Processor -- start 2024-01-15T14:35:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.057+0000 7fe3b20d0700 1 -- start start 2024-01-15T14:35:06.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3b20d0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3ac161690 0x7fe3ac0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3b20d0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac09b110 0x7fe3ac0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3b20d0700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac0d0a70 con 0x7fe3ac161a70 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3b20d0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac0d1c10 con 0x7fe3ac161690 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3b20d0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0b2ab0 con 0x7fe3ac09b110 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3ac161690 0x7fe3ac0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3ac161690 0x7fe3ac0d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56214/0 (socket says 172.21.15.77:56214) 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 -- 172.21.15.77:0/1049392830 learned_addr learned my addr 172.21.15.77:0/1049392830 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3abfff700 1 --2- 172.21.15.77:0/1049392830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac09b110 0x7fe3ac0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 -- 172.21.15.77:0/1049392830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac09b110 msgr2=0x7fe3ac0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 --2- 172.21.15.77:0/1049392830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac09b110 0x7fe3ac0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 -- 172.21.15.77:0/1049392830 >> v1:172.21.15.159:6789/0 conn(0x7fe3ac161a70 legacy=0x7fe3ac09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 -- 172.21.15.77:0/1049392830 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac0b2d90 con 0x7fe3ac161690 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3ab7fe700 1 --2- 172.21.15.77:0/1049392830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161690 0x7fe3ac0d0f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00098f0 tx=0x7fe3a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a9637f66e3c7dc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.058+0000 7fe3aa7fc700 1 -- 172.21.15.77:0/1049392830 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0011830 con 0x7fe3ac161690 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3aa7fc700 1 -- 172.21.15.77:0/1049392830 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a0011990 con 0x7fe3ac161690 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3aa7fc700 1 -- 172.21.15.77:0/1049392830 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0011d20 con 0x7fe3ac161690 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 -- 172.21.15.77:0/1049392830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161690 msgr2=0x7fe3ac0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/1049392830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161690 0x7fe3ac0d0f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00098f0 tx=0x7fe3a0009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 -- 172.21.15.77:0/1049392830 shutdown_connections 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/1049392830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac09b110 0x7fe3ac0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/1049392830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161690 0x7fe3ac0d0f20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 -- 172.21.15.77:0/1049392830 >> 172.21.15.77:0/1049392830 conn(0x7fe3ac0d1e50 msgr2=0x7fe3ac0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 -- 172.21.15.77:0/1049392830 shutdown_connections 2024-01-15T14:35:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.059+0000 7fe3b20d0700 1 -- 172.21.15.77:0/1049392830 wait complete. 2024-01-15T14:35:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.060+0000 7fe3b20d0700 1 Processor -- start 2024-01-15T14:35:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.060+0000 7fe3b20d0700 1 -- start start 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3b20d0700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac09b110 0x7fe3ac0c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3b20d0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac161690 0x7fe3ac1470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3b20d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 0x7fe3ac14add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3b20d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15efd0 con 0x7fe3ac161a70 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3b20d0700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac1475e0 con 0x7fe3ac09b110 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3b20d0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac147880 con 0x7fe3ac161690 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 0x7fe3ac14add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 0x7fe3ac14add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56218/0 (socket says 172.21.15.77:56218) 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3abfff700 1 -- 172.21.15.77:0/3480769996 learned_addr learned my addr 172.21.15.77:0/3480769996 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3ab7fe700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac09b110 0x7fe3ac0c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.061+0000 7fe3aaffd700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac161690 0x7fe3ac1470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3abfff700 1 -- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac161690 msgr2=0x7fe3ac1470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3abfff700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac161690 0x7fe3ac1470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3abfff700 1 -- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac09b110 msgr2=0x7fe3ac0c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3abfff700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac09b110 0x7fe3ac0c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3abfff700 1 -- 172.21.15.77:0/3480769996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0009550 con 0x7fe3ac161a70 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3abfff700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 0x7fe3ac14add0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400efc0 tx=0x7fe3a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4009ca0 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a400f040 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4014480 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac147b60 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.062+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac15f1e0 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.063+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac0d1710 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.063+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe3a40145e0 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.064+0000 7fe393fff700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe39806b350 0x7fe39806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.064+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3a408a8d0 con 0x7fe3ac161a70 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.064+0000 7fe3ab7fe700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe39806b350 0x7fe39806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.064+0000 7fe3ab7fe700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe39806b350 0x7fe39806d810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009c40 tx=0x7fe3a0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:06.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.068+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3ac0d1710 con 0x7fe3ac161a70 2024-01-15T14:35:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:06.308+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe3ac0d1710 con 0x7fe3ac161a70 2024-01-15T14:35:07.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.228+0000 7fe393fff700 1 -- 172.21.15.77:0/3480769996 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe3ac0d1710 con 0x7fe3ac161a70 2024-01-15T14:35:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe39806b350 msgr2=0x7fe39806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe39806b350 0x7fe39806d810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009c40 tx=0x7fe3a0008030 comp rx=0 tx=0).stop 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 msgr2=0x7fe3ac14add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 0x7fe3ac14add0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400efc0 tx=0x7fe3a400c3a0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 shutdown_connections 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe39806b350 0x7fe39806d810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3ac161a70 0x7fe3ac14add0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe3ac09b110 0x7fe3ac0c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 --2- 172.21.15.77:0/3480769996 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe3ac161690 0x7fe3ac1470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 >> 172.21.15.77:0/3480769996 conn(0x7fe3ac0d1e50 msgr2=0x7fe3ac061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 shutdown_connections 2024-01-15T14:35:07.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.231+0000 7fe3b20d0700 1 -- 172.21.15.77:0/3480769996 wait complete. 2024-01-15T14:35:07.246 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T14:35:07.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.376+0000 7f20a9794700 1 Processor -- start 2024-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.376+0000 7f20a9794700 1 -- start start 2024-01-15T14:35:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a9794700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20a415e2c0 0x7f20a40d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a9794700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f20a40b2880 0x7f20a40d2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a9794700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a406b840 con 0x7f20a415dee0 2024-01-15T14:35:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a9794700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a406c3c0 con 0x7f20a415e2c0 2024-01-15T14:35:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a9794700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a406bb20 con 0x7f20a40b2880 2024-01-15T14:35:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20a415e2c0 0x7f20a40d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20a415e2c0 0x7f20a40d1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56222/0 (socket says 172.21.15.77:56222) 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 -- 172.21.15.77:0/3527474044 learned_addr learned my addr 172.21.15.77:0/3527474044 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 -- 172.21.15.77:0/3527474044 >> v2:172.21.15.159:3300/0 conn(0x7f20a40b2880 msgr2=0x7f20a40d2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a37fe700 1 --2- 172.21.15.77:0/3527474044 >> v2:172.21.15.159:3300/0 conn(0x7f20a40b2880 0x7f20a40d2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 --2- 172.21.15.77:0/3527474044 >> v2:172.21.15.159:3300/0 conn(0x7f20a40b2880 0x7f20a40d2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 -- 172.21.15.77:0/3527474044 >> v1:172.21.15.159:6789/0 conn(0x7f20a415dee0 legacy=0x7f20a406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.377+0000 7f20a27fc700 1 -- 172.21.15.77:0/3527474044 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a406be00 con 0x7f20a415e2c0 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a27fc700 1 --2- 172.21.15.77:0/3527474044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415e2c0 0x7f20a40d1c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f20a406d020 tx=0x7f2098009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1740c144dd83f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a1ffb700 1 -- 172.21.15.77:0/3527474044 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098004230 con 0x7f20a415e2c0 2024-01-15T14:35:07.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a1ffb700 1 -- 172.21.15.77:0/3527474044 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098004390 con 0x7f20a415e2c0 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a1ffb700 1 -- 172.21.15.77:0/3527474044 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980036a0 con 0x7f20a415e2c0 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 -- 172.21.15.77:0/3527474044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415e2c0 msgr2=0x7f20a40d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 --2- 172.21.15.77:0/3527474044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415e2c0 0x7f20a40d1c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f20a406d020 tx=0x7f2098009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 -- 172.21.15.77:0/3527474044 shutdown_connections 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 --2- 172.21.15.77:0/3527474044 >> v2:172.21.15.159:3300/0 conn(0x7f20a40b2880 0x7f20a40d2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 --2- 172.21.15.77:0/3527474044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415e2c0 0x7f20a40d1c40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 -- 172.21.15.77:0/3527474044 >> 172.21.15.77:0/3527474044 conn(0x7f20a4073130 msgr2=0x7f20a409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 -- 172.21.15.77:0/3527474044 shutdown_connections 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.378+0000 7f20a9794700 1 -- 172.21.15.77:0/3527474044 wait complete. 2024-01-15T14:35:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 Processor -- start 2024-01-15T14:35:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 -- start start 2024-01-15T14:35:07.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a40b2880 0x7f20a409f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:07.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 0x7f20a40af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:07.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a415e2c0 0x7f20a40afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:07.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a414fd50 con 0x7f20a415dee0 2024-01-15T14:35:07.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4150030 con 0x7f20a40b2880 2024-01-15T14:35:07.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.380+0000 7f20a9794700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4150310 con 0x7f20a415e2c0 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 0x7f20a40af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 0x7f20a40af7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56238/0 (socket says 172.21.15.77:56238) 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 -- 172.21.15.77:0/1087473348 learned_addr learned my addr 172.21.15.77:0/1087473348 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a37fe700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a415e2c0 0x7f20a40afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a2ffd700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a40b2880 0x7f20a409f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 -- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a415e2c0 msgr2=0x7f20a40afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a415e2c0 0x7f20a40afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 -- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a40b2880 msgr2=0x7f20a409f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a40b2880 0x7f20a409f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:07.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 -- 172.21.15.77:0/1087473348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2098009550 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a37fe700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a415e2c0 0x7f20a40afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a27fc700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 0x7f20a40af7c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2098009130 tx=0x7f2098004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098011630 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a41505f0 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098011790 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098011b40 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.381+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a409fa80 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.382+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a4068450 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.383+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2098011ca0 con 0x7f20a415dee0 2024-01-15T14:35:07.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.383+0000 7f208b7fe700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f209006b350 0x7f209006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:07.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.383+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f209808b220 con 0x7f20a415dee0 2024-01-15T14:35:07.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.383+0000 7f20a2ffd700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f209006b350 0x7f209006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:07.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.383+0000 7f20a2ffd700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f209006b350 0x7f209006d810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20a40c8c60 tx=0x7f2094009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:07.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.388+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20980560a0 con 0x7f20a415dee0 2024-01-15T14:35:07.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:07.622+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f20a4069020 con 0x7f20a415dee0 2024-01-15T14:35:08.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.229+0000 7f208b7fe700 1 -- 172.21.15.77:0/1087473348 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f20980596c0 con 0x7f20a415dee0 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f209006b350 msgr2=0x7f209006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f209006b350 0x7f209006d810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20a40c8c60 tx=0x7f2094009040 comp rx=0 tx=0).stop 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 msgr2=0x7f20a40af7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 0x7f20a40af7c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2098009130 tx=0x7f2098004750 comp rx=0 tx=0).stop 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 shutdown_connections 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f209006b350 0x7f209006d810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f20a415e2c0 0x7f20a40afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f20a40b2880 0x7f20a409f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 --2- 172.21.15.77:0/1087473348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a415dee0 0x7f20a40af7c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 >> 172.21.15.77:0/1087473348 conn(0x7f20a4073130 msgr2=0x7f20a406fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.231+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 shutdown_connections 2024-01-15T14:35:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.232+0000 7f20a9794700 1 -- 172.21.15.77:0/1087473348 wait complete. 2024-01-15T14:35:08.245 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T14:35:08.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:08.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.370+0000 7f15174c5700 1 Processor -- start 2024-01-15T14:35:08.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.370+0000 7f15174c5700 1 -- start start 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f15174c5700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f151015e2a0 0x7f151006ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f15174c5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ce090 0x7f151006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f15174c5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1510073f90 0x7f1510074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f15174c5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1510074960 con 0x7f1510073f90 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f15174c5700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151006e7f0 con 0x7f151015e2a0 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f15174c5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151006dec0 con 0x7f15100ce090 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f1515a62700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1510073f90 0x7f1510074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f1515a62700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1510073f90 0x7f1510074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56246/0 (socket says 172.21.15.77:56246) 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.371+0000 7f1515a62700 1 -- 172.21.15.77:0/1565366180 learned_addr learned my addr 172.21.15.77:0/1565366180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515261700 1 --2- 172.21.15.77:0/1565366180 >> v2:172.21.15.159:3300/0 conn(0x7f151015e2a0 0x7f151006ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1514a60700 1 --2- 172.21.15.77:0/1565366180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ce090 0x7f151006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515a62700 1 -- 172.21.15.77:0/1565366180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ce090 msgr2=0x7f151006ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515a62700 1 --2- 172.21.15.77:0/1565366180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ce090 0x7f151006ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515a62700 1 -- 172.21.15.77:0/1565366180 >> v2:172.21.15.159:3300/0 conn(0x7f151015e2a0 msgr2=0x7f151006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515a62700 1 --2- 172.21.15.77:0/1565366180 >> v2:172.21.15.159:3300/0 conn(0x7f151015e2a0 0x7f151006ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515a62700 1 -- 172.21.15.77:0/1565366180 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151006e1a0 con 0x7f1510073f90 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515261700 1 --2- 172.21.15.77:0/1565366180 >> v2:172.21.15.159:3300/0 conn(0x7f151015e2a0 0x7f151006ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1515a62700 1 --2- 172.21.15.77:0/1565366180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1510073f90 0x7f1510074420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f151006d0c0 tx=0x7f150800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aaad7b7c1c6f733 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1507fff700 1 -- 172.21.15.77:0/1565366180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150800e070 con 0x7f1510073f90 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1507fff700 1 -- 172.21.15.77:0/1565366180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150800b9c0 con 0x7f1510073f90 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f1507fff700 1 -- 172.21.15.77:0/1565366180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150800bd70 con 0x7f1510073f90 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f15174c5700 1 -- 172.21.15.77:0/1565366180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1510073f90 msgr2=0x7f1510074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.372+0000 7f15174c5700 1 --2- 172.21.15.77:0/1565366180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1510073f90 0x7f1510074420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f151006d0c0 tx=0x7f150800b330 comp rx=0 tx=0).stop 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 -- 172.21.15.77:0/1565366180 shutdown_connections 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 --2- 172.21.15.77:0/1565366180 >> v2:172.21.15.159:3300/0 conn(0x7f151015e2a0 0x7f151006ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 --2- 172.21.15.77:0/1565366180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1510073f90 0x7f1510074420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 --2- 172.21.15.77:0/1565366180 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15100ce090 0x7f151006ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 -- 172.21.15.77:0/1565366180 >> 172.21.15.77:0/1565366180 conn(0x7f15100cd570 msgr2=0x7f15100cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 -- 172.21.15.77:0/1565366180 shutdown_connections 2024-01-15T14:35:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.373+0000 7f15174c5700 1 -- 172.21.15.77:0/1565366180 wait complete. 2024-01-15T14:35:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.374+0000 7f15174c5700 1 Processor -- start 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.374+0000 7f15174c5700 1 -- start start 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.374+0000 7f15174c5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1510073f90 0x7f15101565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.374+0000 7f15174c5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15100ce090 0x7f1510158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.374+0000 7f15174c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 0x7f15100ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f15174c5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510158d10 con 0x7f151015e2a0 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f15174c5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510158ff0 con 0x7f15100ce090 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f15174c5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15101592d0 con 0x7f1510073f90 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 0x7f15100ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 0x7f15100ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56258/0 (socket says 172.21.15.77:56258) 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 -- 172.21.15.77:0/1995802679 learned_addr learned my addr 172.21.15.77:0/1995802679 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515261700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1510073f90 0x7f15101565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1514a60700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15100ce090 0x7f1510158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 -- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1510073f90 msgr2=0x7f15101565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1510073f90 0x7f15101565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 -- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15100ce090 msgr2=0x7f1510158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15100ce090 0x7f1510158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 -- 172.21.15.77:0/1995802679 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1508009b30 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1515a62700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 0x7f15100ae8c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1510073510 tx=0x7f1508004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15080099b0 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1508004250 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15101595b0 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508004600 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.375+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510156b10 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.376+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15100cdeb0 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.377+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1508004760 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.377+0000 7f1505ffb700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f14f406b350 0x7f14f406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.377+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f150808abd0 con 0x7f151015e2a0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.377+0000 7f1515261700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f14f406b350 0x7f14f406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.378+0000 7f1515261700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f14f406b350 0x7f14f406d810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1500006f10 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:08.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.382+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1508055ad0 con 0x7f151015e2a0 2024-01-15T14:35:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.612+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1510156ca0 con 0x7f151015e2a0 2024-01-15T14:35:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.613+0000 7f1505ffb700 1 -- 172.21.15.77:0/1995802679 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f150801f020 con 0x7f151015e2a0 2024-01-15T14:35:08.613 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:08.613 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f14f406b350 msgr2=0x7f14f406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f14f406b350 0x7f14f406d810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1500006f10 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 msgr2=0x7f15100ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 0x7f15100ae8c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1510073510 tx=0x7f1508004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 shutdown_connections 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f14f406b350 0x7f14f406d810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1510073f90 0x7f15101565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151015e2a0 0x7f15100ae8c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 --2- 172.21.15.77:0/1995802679 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15100ce090 0x7f1510158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 >> 172.21.15.77:0/1995802679 conn(0x7f15100cd570 msgr2=0x7f151006b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 shutdown_connections 2024-01-15T14:35:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.615+0000 7f15174c5700 1 -- 172.21.15.77:0/1995802679 wait complete. 2024-01-15T14:35:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:08.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:08.669 INFO:tasks.ceph.mds.d.smithi159.stderr:2024-01-15T14:35:08.669+0000 7f0c1a740700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T14:35:08.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.747+0000 7f82fbad2700 1 Processor -- start 2024-01-15T14:35:08.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.747+0000 7f82fbad2700 1 -- start start 2024-01-15T14:35:08.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82fbad2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 0x7f82f41603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82fbad2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40cd100 con 0x7f82f40d2160 2024-01-15T14:35:08.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82fbad2700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40d09a0 con 0x7f82f415bed0 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82fbad2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b1ea0 con 0x7f82f40b19f0 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f906d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f82f40d2160 0x7f82f40cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48508/0 (socket says 172.21.15.77:48508) 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f906d700 1 -- 172.21.15.77:0/136531376 learned_addr learned my addr 172.21.15.77:0/136531376 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82fa06f700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 0x7f82f41603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523299961 0 0) 0x7f82f40cd100 con 0x7f82f40d2160 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e00035b0 con 0x7f82f40d2160 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 221442276 0 0) 0x7f82e00035b0 con 0x7f82f40d2160 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 msgr2=0x7f82f41603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 0x7f82f41603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 >> v1:172.21.15.159:6789/0 conn(0x7f82f415bed0 legacy=0x7f82f40d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40cc7d0 con 0x7f82f40d2160 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f82f0001670 con 0x7f82f40d2160 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 >> v1:172.21.15.77:6789/0 conn(0x7f82f40d2160 legacy=0x7f82f40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e0003a30 0x7f82e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 0x7f82e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e0004d90 0x7f82e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b1ea0 con 0x7f82e0004d90 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e000ba10 con 0x7f82e0003a30 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e000bbd0 con 0x7f82f40b19f0 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82fa06f700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e0004d90 0x7f82e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f986e700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e0003a30 0x7f82e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.749+0000 7f82f906d700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 0x7f82e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 -- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 msgr2=0x7f82e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b19f0 0x7f82e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 -- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e0003a30 msgr2=0x7f82e0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82e0003a30 0x7f82e0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 -- 172.21.15.77:0/136531376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00035b0 con 0x7f82e0004d90 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e0004d90 0x7f82e0005210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009fc0 tx=0x7f82e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c316ffdd29f0a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00ec30 con 0x7f82e0004d90 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec00f040 con 0x7f82e0004d90 2024-01-15T14:35:08.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82f886c700 1 -- 172.21.15.77:0/136531376 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec014480 con 0x7f82e0004d90 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fbad2700 1 -- 172.21.15.77:0/136531376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e0004d90 msgr2=0x7f82e0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fbad2700 1 --2- 172.21.15.77:0/136531376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82e0004d90 0x7f82e0005210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009fc0 tx=0x7f82e0005880 comp rx=0 tx=0).stop 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fa06f700 1 -- 172.21.15.77:0/136531376 reap_dead start 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fbad2700 1 -- 172.21.15.77:0/136531376 shutdown_connections 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.750+0000 7f82fbad2700 1 -- 172.21.15.77:0/136531376 >> 172.21.15.77:0/136531376 conn(0x7f82f4073130 msgr2=0x7f82f409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.751+0000 7f82fbad2700 1 -- 172.21.15.77:0/136531376 shutdown_connections 2024-01-15T14:35:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.751+0000 7f82fbad2700 1 -- 172.21.15.77:0/136531376 wait complete. 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.752+0000 7f82fbad2700 1 Processor -- start 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.752+0000 7f82fbad2700 1 -- start start 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fbad2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 0x7f82f41f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fbad2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b67a0 0x7f82f40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fbad2700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f40b4d30 0x7f82f414af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fbad2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b51b0 con 0x7f82f415ecf0 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fbad2700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b7900 con 0x7f82f40b4d30 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fbad2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41f5d90 con 0x7f82f40b67a0 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 0x7f82f41f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 0x7f82f41f5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56282/0 (socket says 172.21.15.77:56282) 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 -- 172.21.15.77:0/1130716879 learned_addr learned my addr 172.21.15.77:0/1130716879 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:08.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f906d700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b67a0 0x7f82f40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fa06f700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f40b4d30 0x7f82f414af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 -- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b67a0 msgr2=0x7f82f40b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b67a0 0x7f82f40b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 -- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f40b4d30 msgr2=0x7f82f414af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f40b4d30 0x7f82f414af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 -- 172.21.15.77:0/1130716879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40b6cf0 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82fa06f700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f40b4d30 0x7f82f414af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f906d700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b67a0 0x7f82f40b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.753+0000 7f82f986e700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 0x7f82f41f5850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f82e400f850 tx=0x7f82e400fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.754+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e400fe50 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.754+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e40052e0 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.754+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4003680 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.754+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f40b6fd0 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.754+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40b7590 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.755+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f40b7750 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.755+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f82e40037e0 con 0x7f82f415ecf0 2024-01-15T14:35:08.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.755+0000 7f82ea7fc700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f82d806b350 0x7f82d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:08.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.756+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82e4089bb0 con 0x7f82f415ecf0 2024-01-15T14:35:08.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.756+0000 7f82f906d700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f82d806b350 0x7f82d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:08.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.756+0000 7f82f906d700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f82d806b350 0x7f82d806d810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f82f414e9f0 tx=0x7f82f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:08.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.760+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82e4058410 con 0x7f82f415ecf0 2024-01-15T14:35:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.950+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f4067f60 con 0x7f82f415ecf0 2024-01-15T14:35:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.951+0000 7f82ea7fc700 1 -- 172.21.15.77:0/1130716879 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f82e4057fa0 con 0x7f82f415ecf0 2024-01-15T14:35:08.951 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:08.952 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f82d806b350 msgr2=0x7f82d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f82d806b350 0x7f82d806d810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f82f414e9f0 tx=0x7f82f0009040 comp rx=0 tx=0).stop 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 msgr2=0x7f82f41f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 0x7f82f41f5850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f82e400f850 tx=0x7f82e400fc10 comp rx=0 tx=0).stop 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 shutdown_connections 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f82d806b350 0x7f82d806d810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f82f40b4d30 0x7f82f414af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f415ecf0 0x7f82f41f5850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 --2- 172.21.15.77:0/1130716879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f82f40b67a0 0x7f82f40b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 >> 172.21.15.77:0/1130716879 conn(0x7f82f4073130 msgr2=0x7f82f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 shutdown_connections 2024-01-15T14:35:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:08.953+0000 7f82fbad2700 1 -- 172.21.15.77:0/1130716879 wait complete. 2024-01-15T14:35:08.965 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T14:35:08.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T14:35:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.086+0000 7f834c4f5700 1 Processor -- start 2024-01-15T14:35:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.086+0000 7f834c4f5700 1 -- start start 2024-01-15T14:35:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.087+0000 7f834c4f5700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f834406c650 0x7f8344165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.087+0000 7f834c4f5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834415f800 con 0x7f834415af10 2024-01-15T14:35:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.087+0000 7f834c4f5700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83441414c0 con 0x7f83440ce140 2024-01-15T14:35:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.087+0000 7f834c4f5700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834415c300 con 0x7f834406c650 2024-01-15T14:35:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834a291700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f834415af10 0x7f834415e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48520/0 (socket says 172.21.15.77:48520) 2024-01-15T14:35:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834a291700 1 -- 172.21.15.77:0/732791147 learned_addr learned my addr 172.21.15.77:0/732791147 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834aa92700 1 --2- 172.21.15.77:0/732791147 >> v2:172.21.15.159:3300/0 conn(0x7f834406c650 0x7f8344165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319701248 0 0) 0x7f834415f800 con 0x7f834415af10 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83300035b0 con 0x7f834415af10 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795554734 0 0) 0x7f83441414c0 con 0x7f83440ce140 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834415f800 con 0x7f83440ce140 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834aa92700 1 -- 172.21.15.77:0/732791147 >> v1:172.21.15.159:6789/0 conn(0x7f83440ce140 legacy=0x7f834415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834aa92700 1 -- 172.21.15.77:0/732791147 >> v1:172.21.15.77:6789/0 conn(0x7f834415af10 legacy=0x7f834415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169410467 0 0) 0x7f83300035b0 con 0x7f834415af10 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834aa92700 1 -- 172.21.15.77:0/732791147 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83440ce9f0 con 0x7f834406c650 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834aa92700 1 --2- 172.21.15.77:0/732791147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f834406c650 0x7f8344165d70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f833c009d90 tx=0x7f833c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c493567237dc516 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00e070 con 0x7f834406c650 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c003a20 con 0x7f834406c650 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.088+0000 7f834928f700 1 -- 172.21.15.77:0/732791147 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334001670 con 0x7f834406c650 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 -- 172.21.15.77:0/732791147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f834406c650 msgr2=0x7f8344165d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 --2- 172.21.15.77:0/732791147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f834406c650 0x7f8344165d70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f833c009d90 tx=0x7f833c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 -- 172.21.15.77:0/732791147 shutdown_connections 2024-01-15T14:35:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 --2- 172.21.15.77:0/732791147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f834406c650 0x7f8344165d70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 -- 172.21.15.77:0/732791147 >> 172.21.15.77:0/732791147 conn(0x7f83440cd570 msgr2=0x7f83440cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 -- 172.21.15.77:0/732791147 shutdown_connections 2024-01-15T14:35:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.089+0000 7f834c4f5700 1 -- 172.21.15.77:0/732791147 wait complete. 2024-01-15T14:35:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.090+0000 7f834c4f5700 1 Processor -- start 2024-01-15T14:35:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.090+0000 7f834c4f5700 1 -- start start 2024-01-15T14:35:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834c4f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 0x7f8344143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834c4f5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440ce140 0x7f834415c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834c4f5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f834415af10 0x7f8344157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834c4f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b4500 con 0x7f834406c650 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834c4f5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83441438c0 con 0x7f83440ce140 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834c4f5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b3e40 con 0x7f834415af10 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 0x7f8344143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 0x7f8344143230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56286/0 (socket says 172.21.15.77:56286) 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 -- 172.21.15.77:0/899676284 learned_addr learned my addr 172.21.15.77:0/899676284 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f8349a90700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440ce140 0x7f834415c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834aa92700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f834415af10 0x7f8344157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 -- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f834415af10 msgr2=0x7f8344157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f834415af10 0x7f8344157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 -- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440ce140 msgr2=0x7f834415c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440ce140 0x7f834415c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 -- 172.21.15.77:0/899676284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c009a70 con 0x7f834406c650 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.091+0000 7f834a291700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 0x7f8344143230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f833400c750 tx=0x7f833400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.092+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004c80 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.092+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8334004de0 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.092+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83440b4120 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.092+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340072d0 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.092+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8344143ba0 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.092+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83440b4640 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.093+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8334007430 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.093+0000 7f833affd700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f832806b350 0x7f832806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.093+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8334089980 con 0x7f834406c650 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.093+0000 7f8349a90700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f832806b350 0x7f832806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.094+0000 7f8349a90700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f832806b350 0x7f832806d810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83440ce540 tx=0x7f8340009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.098+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8334054880 con 0x7f834406c650 2024-01-15T14:35:09.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.288+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83440cefb0 con 0x7f832806b350 2024-01-15T14:35:09.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.296+0000 7f833affd700 1 -- 172.21.15.77:0/899676284 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264671 (secure 0 0 0) 0x7f8340001cd0 con 0x7f832806b350 2024-01-15T14:35:09.299 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f832806b350 msgr2=0x7f832806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f832806b350 0x7f832806d810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83440ce540 tx=0x7f8340009040 comp rx=0 tx=0).stop 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 msgr2=0x7f8344143230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 0x7f8344143230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f833400c750 tx=0x7f833400ca60 comp rx=0 tx=0).stop 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 shutdown_connections 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f832806b350 0x7f832806d810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f834415af10 0x7f8344157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f834406c650 0x7f8344143230 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 --2- 172.21.15.77:0/899676284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f83440ce140 0x7f834415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 >> 172.21.15.77:0/899676284 conn(0x7f83440cd570 msgr2=0x7f834406dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:09.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 shutdown_connections 2024-01-15T14:35:09.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.299+0000 7f834c4f5700 1 -- 172.21.15.77:0/899676284 wait complete. 2024-01-15T14:35:09.302 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-01-15T14:35:09.311 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-15T14:35:09.018704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219056,"kb_used_data":2860,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755664,"statfs":{"total":773094113280,"available":772869799936,"internally_reserved":0,"allocated":2928640,"data_stored":1758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.832059"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223558+0000","last_change":"2024-01-15T14:35:05.242883+0000","last_active":"2024-01-15T14:35:06.223558+0000","last_peered":"2024-01-15T14:35:06.223558+0000","last_clean":"2024-01-15T14:35:06.223558+0000","last_became_active":"2024-01-15T14:35:05.242650+0000","last_became_peered":"2024-01-15T14:35:05.242650+0000","last_unstale":"2024-01-15T14:35:06.223558+0000","last_undegraded":"2024-01-15T14:35:06.223558+0000","last_fullsized":"2024-01-15T14:35:06.223558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:06:13.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038751+0000","last_change":"2024-01-15T14:35:05.230934+0000","last_active":"2024-01-15T14:35:07.038751+0000","last_peered":"2024-01-15T14:35:07.038751+0000","last_clean":"2024-01-15T14:35:07.038751+0000","last_became_active":"2024-01-15T14:35:05.230356+0000","last_became_peered":"2024-01-15T14:35:05.230356+0000","last_unstale":"2024-01-15T14:35:07.038751+0000","last_undegraded":"2024-01-15T14:35:07.038751+0000","last_fullsized":"2024-01-15T14:35:07.038751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914251+0000","last_change":"2024-01-15T14:35:05.228307+0000","last_active":"2024-01-15T14:35:06.914251+0000","last_peered":"2024-01-15T14:35:06.914251+0000","last_clean":"2024-01-15T14:35:06.914251+0000","last_became_active":"2024-01-15T14:35:05.227876+0000","last_became_peered":"2024-01-15T14:35:05.227876+0000","last_unstale":"2024-01-15T14:35:06.914251+0000","last_undegraded":"2024-01-15T14:35:06.914251+0000","last_fullsized":"2024-01-15T14:35:06.914251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:11.269770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338166+0000","last_change":"2024-01-15T14:35:05.236667+0000","last_active":"2024-01-15T14:35:06.338166+0000","last_peered":"2024-01-15T14:35:06.338166+0000","last_clean":"2024-01-15T14:35:06.338166+0000","last_became_active":"2024-01-15T14:35:05.236363+0000","last_became_peered":"2024-01-15T14:35:05.236363+0000","last_unstale":"2024-01-15T14:35:06.338166+0000","last_undegraded":"2024-01-15T14:35:06.338166+0000","last_fullsized":"2024-01-15T14:35:06.338166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222610+0000","last_change":"2024-01-15T14:35:05.239721+0000","last_active":"2024-01-15T14:35:06.222610+0000","last_peered":"2024-01-15T14:35:06.222610+0000","last_clean":"2024-01-15T14:35:06.222610+0000","last_became_active":"2024-01-15T14:35:05.235648+0000","last_became_peered":"2024-01-15T14:35:05.235648+0000","last_unstale":"2024-01-15T14:35:06.222610+0000","last_undegraded":"2024-01-15T14:35:06.222610+0000","last_fullsized":"2024-01-15T14:35:06.222610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337634+0000","last_change":"2024-01-15T14:35:05.230388+0000","last_active":"2024-01-15T14:35:06.337634+0000","last_peered":"2024-01-15T14:35:06.337634+0000","last_clean":"2024-01-15T14:35:06.337634+0000","last_became_active":"2024-01-15T14:35:05.230109+0000","last_became_peered":"2024-01-15T14:35:05.230109+0000","last_unstale":"2024-01-15T14:35:06.337634+0000","last_undegraded":"2024-01-15T14:35:06.337634+0000","last_fullsized":"2024-01-15T14:35:06.337634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222611+0000","last_change":"2024-01-15T14:35:05.239457+0000","last_active":"2024-01-15T14:35:06.222611+0000","last_peered":"2024-01-15T14:35:06.222611+0000","last_clean":"2024-01-15T14:35:06.222611+0000","last_became_active":"2024-01-15T14:35:05.239152+0000","last_became_peered":"2024-01-15T14:35:05.239152+0000","last_unstale":"2024-01-15T14:35:06.222611+0000","last_undegraded":"2024-01-15T14:35:06.222611+0000","last_fullsized":"2024-01-15T14:35:06.222611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220500+0000","last_change":"2024-01-15T14:35:05.219905+0000","last_active":"2024-01-15T14:35:06.220500+0000","last_peered":"2024-01-15T14:35:06.220500+0000","last_clean":"2024-01-15T14:35:06.220500+0000","last_became_active":"2024-01-15T14:35:05.219331+0000","last_became_peered":"2024-01-15T14:35:05.219331+0000","last_unstale":"2024-01-15T14:35:06.220500+0000","last_undegraded":"2024-01-15T14:35:06.220500+0000","last_fullsized":"2024-01-15T14:35:06.220500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223409+0000","last_change":"2024-01-15T14:35:05.232665+0000","last_active":"2024-01-15T14:35:06.223409+0000","last_peered":"2024-01-15T14:35:06.223409+0000","last_clean":"2024-01-15T14:35:06.223409+0000","last_became_active":"2024-01-15T14:35:05.232381+0000","last_became_peered":"2024-01-15T14:35:05.232381+0000","last_unstale":"2024-01-15T14:35:06.223409+0000","last_undegraded":"2024-01-15T14:35:06.223409+0000","last_fullsized":"2024-01-15T14:35:06.223409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222267+0000","last_change":"2024-01-15T14:35:05.222078+0000","last_active":"2024-01-15T14:35:06.222267+0000","last_peered":"2024-01-15T14:35:06.222267+0000","last_clean":"2024-01-15T14:35:06.222267+0000","last_became_active":"2024-01-15T14:35:05.221809+0000","last_became_peered":"2024-01-15T14:35:05.221809+0000","last_unstale":"2024-01-15T14:35:06.222267+0000","last_undegraded":"2024-01-15T14:35:06.222267+0000","last_fullsized":"2024-01-15T14:35:06.222267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221969+0000","last_change":"2024-01-15T14:35:05.222838+0000","last_active":"2024-01-15T14:35:06.221969+0000","last_peered":"2024-01-15T14:35:06.221969+0000","last_clean":"2024-01-15T14:35:06.221969+0000","last_became_active":"2024-01-15T14:35:05.221973+0000","last_became_peered":"2024-01-15T14:35:05.221973+0000","last_unstale":"2024-01-15T14:35:06.221969+0000","last_undegraded":"2024-01-15T14:35:06.221969+0000","last_fullsized":"2024-01-15T14:35:06.221969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038638+0000","last_change":"2024-01-15T14:35:05.242599+0000","last_active":"2024-01-15T14:35:07.038638+0000","last_peered":"2024-01-15T14:35:07.038638+0000","last_clean":"2024-01-15T14:35:07.038638+0000","last_became_active":"2024-01-15T14:35:05.242171+0000","last_became_peered":"2024-01-15T14:35:05.242171+0000","last_unstale":"2024-01-15T14:35:07.038638+0000","last_undegraded":"2024-01-15T14:35:07.038638+0000","last_fullsized":"2024-01-15T14:35:07.038638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220701+0000","last_change":"2024-01-15T14:35:05.224493+0000","last_active":"2024-01-15T14:35:06.220701+0000","last_peered":"2024-01-15T14:35:06.220701+0000","last_clean":"2024-01-15T14:35:06.220701+0000","last_became_active":"2024-01-15T14:35:05.224166+0000","last_became_peered":"2024-01-15T14:35:05.224166+0000","last_unstale":"2024-01-15T14:35:06.220701+0000","last_undegraded":"2024-01-15T14:35:06.220701+0000","last_fullsized":"2024-01-15T14:35:06.220701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914368+0000","last_change":"2024-01-15T14:35:05.229113+0000","last_active":"2024-01-15T14:35:06.914368+0000","last_peered":"2024-01-15T14:35:06.914368+0000","last_clean":"2024-01-15T14:35:06.914368+0000","last_became_active":"2024-01-15T14:35:05.228790+0000","last_became_peered":"2024-01-15T14:35:05.228790+0000","last_unstale":"2024-01-15T14:35:06.914368+0000","last_undegraded":"2024-01-15T14:35:06.914368+0000","last_fullsized":"2024-01-15T14:35:06.914368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038480+0000","last_change":"2024-01-15T14:35:05.236987+0000","last_active":"2024-01-15T14:35:07.038480+0000","last_peered":"2024-01-15T14:35:07.038480+0000","last_clean":"2024-01-15T14:35:07.038480+0000","last_became_active":"2024-01-15T14:35:05.236693+0000","last_became_peered":"2024-01-15T14:35:05.236693+0000","last_unstale":"2024-01-15T14:35:07.038480+0000","last_undegraded":"2024-01-15T14:35:07.038480+0000","last_fullsized":"2024-01-15T14:35:07.038480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337512+0000","last_change":"2024-01-15T14:35:05.229665+0000","last_active":"2024-01-15T14:35:06.337512+0000","last_peered":"2024-01-15T14:35:06.337512+0000","last_clean":"2024-01-15T14:35:06.337512+0000","last_became_active":"2024-01-15T14:35:05.229391+0000","last_became_peered":"2024-01-15T14:35:05.229391+0000","last_unstale":"2024-01-15T14:35:06.337512+0000","last_undegraded":"2024-01-15T14:35:06.337512+0000","last_fullsized":"2024-01-15T14:35:06.337512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338230+0000","last_change":"2024-01-15T14:35:05.232892+0000","last_active":"2024-01-15T14:35:06.338230+0000","last_peered":"2024-01-15T14:35:06.338230+0000","last_clean":"2024-01-15T14:35:06.338230+0000","last_became_active":"2024-01-15T14:35:05.232596+0000","last_became_peered":"2024-01-15T14:35:05.232596+0000","last_unstale":"2024-01-15T14:35:06.338230+0000","last_undegraded":"2024-01-15T14:35:06.338230+0000","last_fullsized":"2024-01-15T14:35:06.338230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420132+0000","last_change":"2024-01-15T14:35:05.224599+0000","last_active":"2024-01-15T14:35:06.420132+0000","last_peered":"2024-01-15T14:35:06.420132+0000","last_clean":"2024-01-15T14:35:06.420132+0000","last_became_active":"2024-01-15T14:35:05.224306+0000","last_became_peered":"2024-01-15T14:35:05.224306+0000","last_unstale":"2024-01-15T14:35:06.420132+0000","last_undegraded":"2024-01-15T14:35:06.420132+0000","last_fullsized":"2024-01-15T14:35:06.420132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:35.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222848+0000","last_change":"2024-01-15T14:35:05.235613+0000","last_active":"2024-01-15T14:35:06.222848+0000","last_peered":"2024-01-15T14:35:06.222848+0000","last_clean":"2024-01-15T14:35:06.222848+0000","last_became_active":"2024-01-15T14:35:05.235298+0000","last_became_peered":"2024-01-15T14:35:05.235298+0000","last_unstale":"2024-01-15T14:35:06.222848+0000","last_undegraded":"2024-01-15T14:35:06.222848+0000","last_fullsized":"2024-01-15T14:35:06.222848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914462+0000","last_change":"2024-01-15T14:35:04.217677+0000","last_active":"2024-01-15T14:35:06.914462+0000","last_peered":"2024-01-15T14:35:06.914462+0000","last_clean":"2024-01-15T14:35:06.914462+0000","last_became_active":"2024-01-15T14:35:04.217361+0000","last_became_peered":"2024-01-15T14:35:04.217361+0000","last_unstale":"2024-01-15T14:35:06.914462+0000","last_undegraded":"2024-01-15T14:35:06.914462+0000","last_fullsized":"2024-01-15T14:35:06.914462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220970+0000","last_change":"2024-01-15T14:35:05.225192+0000","last_active":"2024-01-15T14:35:06.220970+0000","last_peered":"2024-01-15T14:35:06.220970+0000","last_clean":"2024-01-15T14:35:06.220970+0000","last_became_active":"2024-01-15T14:35:05.224634+0000","last_became_peered":"2024-01-15T14:35:05.224634+0000","last_unstale":"2024-01-15T14:35:06.220970+0000","last_undegraded":"2024-01-15T14:35:06.220970+0000","last_fullsized":"2024-01-15T14:35:06.220970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220874+0000","last_change":"2024-01-15T14:35:04.204838+0000","last_active":"2024-01-15T14:35:06.220874+0000","last_peered":"2024-01-15T14:35:06.220874+0000","last_clean":"2024-01-15T14:35:06.220874+0000","last_became_active":"2024-01-15T14:35:04.203651+0000","last_became_peered":"2024-01-15T14:35:04.203651+0000","last_unstale":"2024-01-15T14:35:06.220874+0000","last_undegraded":"2024-01-15T14:35:06.220874+0000","last_fullsized":"2024-01-15T14:35:06.220874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914437+0000","last_change":"2024-01-15T14:35:05.228513+0000","last_active":"2024-01-15T14:35:06.914437+0000","last_peered":"2024-01-15T14:35:06.914437+0000","last_clean":"2024-01-15T14:35:06.914437+0000","last_became_active":"2024-01-15T14:35:05.228229+0000","last_became_peered":"2024-01-15T14:35:05.228229+0000","last_unstale":"2024-01-15T14:35:06.914437+0000","last_undegraded":"2024-01-15T14:35:06.914437+0000","last_fullsized":"2024-01-15T14:35:06.914437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223203+0000","last_change":"2024-01-15T14:35:04.217602+0000","last_active":"2024-01-15T14:35:06.223203+0000","last_peered":"2024-01-15T14:35:06.223203+0000","last_clean":"2024-01-15T14:35:06.223203+0000","last_became_active":"2024-01-15T14:35:04.217271+0000","last_became_peered":"2024-01-15T14:35:04.217271+0000","last_unstale":"2024-01-15T14:35:06.223203+0000","last_undegraded":"2024-01-15T14:35:06.223203+0000","last_fullsized":"2024-01-15T14:35:06.223203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:24.748909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337912+0000","last_change":"2024-01-15T14:35:05.234884+0000","last_active":"2024-01-15T14:35:06.337912+0000","last_peered":"2024-01-15T14:35:06.337912+0000","last_clean":"2024-01-15T14:35:06.337912+0000","last_became_active":"2024-01-15T14:35:05.234556+0000","last_became_peered":"2024-01-15T14:35:05.234556+0000","last_unstale":"2024-01-15T14:35:06.337912+0000","last_undegraded":"2024-01-15T14:35:06.337912+0000","last_fullsized":"2024-01-15T14:35:06.337912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222800+0000","last_change":"2024-01-15T14:35:04.217549+0000","last_active":"2024-01-15T14:35:06.222800+0000","last_peered":"2024-01-15T14:35:06.222800+0000","last_clean":"2024-01-15T14:35:06.222800+0000","last_became_active":"2024-01-15T14:35:04.217245+0000","last_became_peered":"2024-01-15T14:35:04.217245+0000","last_unstale":"2024-01-15T14:35:06.222800+0000","last_undegraded":"2024-01-15T14:35:06.222800+0000","last_fullsized":"2024-01-15T14:35:06.222800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337934+0000","last_change":"2024-01-15T14:35:05.230835+0000","last_active":"2024-01-15T14:35:06.337934+0000","last_peered":"2024-01-15T14:35:06.337934+0000","last_clean":"2024-01-15T14:35:06.337934+0000","last_became_active":"2024-01-15T14:35:05.230395+0000","last_became_peered":"2024-01-15T14:35:05.230395+0000","last_unstale":"2024-01-15T14:35:06.337934+0000","last_undegraded":"2024-01-15T14:35:06.337934+0000","last_fullsized":"2024-01-15T14:35:06.337934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220762+0000","last_change":"2024-01-15T14:35:04.216108+0000","last_active":"2024-01-15T14:35:06.220762+0000","last_peered":"2024-01-15T14:35:06.220762+0000","last_clean":"2024-01-15T14:35:06.220762+0000","last_became_active":"2024-01-15T14:35:04.215774+0000","last_became_peered":"2024-01-15T14:35:04.215774+0000","last_unstale":"2024-01-15T14:35:06.220762+0000","last_undegraded":"2024-01-15T14:35:06.220762+0000","last_fullsized":"2024-01-15T14:35:06.220762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038274+0000","last_change":"2024-01-15T14:35:05.243121+0000","last_active":"2024-01-15T14:35:07.038274+0000","last_peered":"2024-01-15T14:35:07.038274+0000","last_clean":"2024-01-15T14:35:07.038274+0000","last_became_active":"2024-01-15T14:35:05.242858+0000","last_became_peered":"2024-01-15T14:35:05.242858+0000","last_unstale":"2024-01-15T14:35:07.038274+0000","last_undegraded":"2024-01-15T14:35:07.038274+0000","last_fullsized":"2024-01-15T14:35:07.038274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:38.527435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222109+0000","last_change":"2024-01-15T14:35:04.214194+0000","last_active":"2024-01-15T14:35:06.222109+0000","last_peered":"2024-01-15T14:35:06.222109+0000","last_clean":"2024-01-15T14:35:06.222109+0000","last_became_active":"2024-01-15T14:35:04.213952+0000","last_became_peered":"2024-01-15T14:35:04.213952+0000","last_unstale":"2024-01-15T14:35:06.222109+0000","last_undegraded":"2024-01-15T14:35:06.222109+0000","last_fullsized":"2024-01-15T14:35:06.222109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914552+0000","last_change":"2024-01-15T14:35:05.237633+0000","last_active":"2024-01-15T14:35:06.914552+0000","last_peered":"2024-01-15T14:35:06.914552+0000","last_clean":"2024-01-15T14:35:06.914552+0000","last_became_active":"2024-01-15T14:35:05.234559+0000","last_became_peered":"2024-01-15T14:35:05.234559+0000","last_unstale":"2024-01-15T14:35:06.914552+0000","last_undegraded":"2024-01-15T14:35:06.914552+0000","last_fullsized":"2024-01-15T14:35:06.914552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220477+0000","last_change":"2024-01-15T14:35:04.212481+0000","last_active":"2024-01-15T14:35:06.220477+0000","last_peered":"2024-01-15T14:35:06.220477+0000","last_clean":"2024-01-15T14:35:06.220477+0000","last_became_active":"2024-01-15T14:35:04.212205+0000","last_became_peered":"2024-01-15T14:35:04.212205+0000","last_unstale":"2024-01-15T14:35:06.220477+0000","last_undegraded":"2024-01-15T14:35:06.220477+0000","last_fullsized":"2024-01-15T14:35:06.220477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220657+0000","last_change":"2024-01-15T14:35:05.229905+0000","last_active":"2024-01-15T14:35:06.220657+0000","last_peered":"2024-01-15T14:35:06.220657+0000","last_clean":"2024-01-15T14:35:06.220657+0000","last_became_active":"2024-01-15T14:35:05.229591+0000","last_became_peered":"2024-01-15T14:35:05.229591+0000","last_unstale":"2024-01-15T14:35:06.220657+0000","last_undegraded":"2024-01-15T14:35:06.220657+0000","last_fullsized":"2024-01-15T14:35:06.220657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337590+0000","last_change":"2024-01-15T14:35:04.212238+0000","last_active":"2024-01-15T14:35:06.337590+0000","last_peered":"2024-01-15T14:35:06.337590+0000","last_clean":"2024-01-15T14:35:06.337590+0000","last_became_active":"2024-01-15T14:35:04.211931+0000","last_became_peered":"2024-01-15T14:35:04.211931+0000","last_unstale":"2024-01-15T14:35:06.337590+0000","last_undegraded":"2024-01-15T14:35:06.337590+0000","last_fullsized":"2024-01-15T14:35:06.337590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:39.371648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038653+0000","last_change":"2024-01-15T14:35:05.215883+0000","last_active":"2024-01-15T14:35:07.038653+0000","last_peered":"2024-01-15T14:35:07.038653+0000","last_clean":"2024-01-15T14:35:07.038653+0000","last_became_active":"2024-01-15T14:35:05.215584+0000","last_became_peered":"2024-01-15T14:35:05.215584+0000","last_unstale":"2024-01-15T14:35:07.038653+0000","last_undegraded":"2024-01-15T14:35:07.038653+0000","last_fullsized":"2024-01-15T14:35:07.038653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222538+0000","last_change":"2024-01-15T14:35:04.212959+0000","last_active":"2024-01-15T14:35:06.222538+0000","last_peered":"2024-01-15T14:35:06.222538+0000","last_clean":"2024-01-15T14:35:06.222538+0000","last_became_active":"2024-01-15T14:35:04.212632+0000","last_became_peered":"2024-01-15T14:35:04.212632+0000","last_unstale":"2024-01-15T14:35:06.222538+0000","last_undegraded":"2024-01-15T14:35:06.222538+0000","last_fullsized":"2024-01-15T14:35:06.222538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337983+0000","last_change":"2024-01-15T14:35:05.232646+0000","last_active":"2024-01-15T14:35:06.337983+0000","last_peered":"2024-01-15T14:35:06.337983+0000","last_clean":"2024-01-15T14:35:06.337983+0000","last_became_active":"2024-01-15T14:35:05.232360+0000","last_became_peered":"2024-01-15T14:35:05.232360+0000","last_unstale":"2024-01-15T14:35:06.337983+0000","last_undegraded":"2024-01-15T14:35:06.337983+0000","last_fullsized":"2024-01-15T14:35:06.337983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337788+0000","last_change":"2024-01-15T14:35:04.207794+0000","last_active":"2024-01-15T14:35:06.337788+0000","last_peered":"2024-01-15T14:35:06.337788+0000","last_clean":"2024-01-15T14:35:06.337788+0000","last_became_active":"2024-01-15T14:35:04.207503+0000","last_became_peered":"2024-01-15T14:35:04.207503+0000","last_unstale":"2024-01-15T14:35:06.337788+0000","last_undegraded":"2024-01-15T14:35:06.337788+0000","last_fullsized":"2024-01-15T14:35:06.337788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914599+0000","last_change":"2024-01-15T14:35:05.228000+0000","last_active":"2024-01-15T14:35:06.914599+0000","last_peered":"2024-01-15T14:35:06.914599+0000","last_clean":"2024-01-15T14:35:06.914599+0000","last_became_active":"2024-01-15T14:35:05.227713+0000","last_became_peered":"2024-01-15T14:35:05.227713+0000","last_unstale":"2024-01-15T14:35:06.914599+0000","last_undegraded":"2024-01-15T14:35:06.914599+0000","last_fullsized":"2024-01-15T14:35:06.914599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038760+0000","last_change":"2024-01-15T14:35:04.212694+0000","last_active":"2024-01-15T14:35:07.038760+0000","last_peered":"2024-01-15T14:35:07.038760+0000","last_clean":"2024-01-15T14:35:07.038760+0000","last_became_active":"2024-01-15T14:35:04.212368+0000","last_became_peered":"2024-01-15T14:35:04.212368+0000","last_unstale":"2024-01-15T14:35:07.038760+0000","last_undegraded":"2024-01-15T14:35:07.038760+0000","last_fullsized":"2024-01-15T14:35:07.038760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038591+0000","last_change":"2024-01-15T14:35:05.235186+0000","last_active":"2024-01-15T14:35:07.038591+0000","last_peered":"2024-01-15T14:35:07.038591+0000","last_clean":"2024-01-15T14:35:07.038591+0000","last_became_active":"2024-01-15T14:35:05.234887+0000","last_became_peered":"2024-01-15T14:35:05.234887+0000","last_unstale":"2024-01-15T14:35:07.038591+0000","last_undegraded":"2024-01-15T14:35:07.038591+0000","last_fullsized":"2024-01-15T14:35:07.038591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:43.060852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222002+0000","last_change":"2024-01-15T14:35:04.212306+0000","last_active":"2024-01-15T14:35:06.222002+0000","last_peered":"2024-01-15T14:35:06.222002+0000","last_clean":"2024-01-15T14:35:06.222002+0000","last_became_active":"2024-01-15T14:35:04.211871+0000","last_became_peered":"2024-01-15T14:35:04.211871+0000","last_unstale":"2024-01-15T14:35:06.222002+0000","last_undegraded":"2024-01-15T14:35:06.222002+0000","last_fullsized":"2024-01-15T14:35:06.222002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223019+0000","last_change":"2024-01-15T14:35:05.242305+0000","last_active":"2024-01-15T14:35:06.223019+0000","last_peered":"2024-01-15T14:35:06.223019+0000","last_clean":"2024-01-15T14:35:06.223019+0000","last_became_active":"2024-01-15T14:35:05.242026+0000","last_became_peered":"2024-01-15T14:35:05.242026+0000","last_unstale":"2024-01-15T14:35:06.223019+0000","last_undegraded":"2024-01-15T14:35:06.223019+0000","last_fullsized":"2024-01-15T14:35:06.223019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914403+0000","last_change":"2024-01-15T14:35:04.205419+0000","last_active":"2024-01-15T14:35:06.914403+0000","last_peered":"2024-01-15T14:35:06.914403+0000","last_clean":"2024-01-15T14:35:06.914403+0000","last_became_active":"2024-01-15T14:35:04.205106+0000","last_became_peered":"2024-01-15T14:35:04.205106+0000","last_unstale":"2024-01-15T14:35:06.914403+0000","last_undegraded":"2024-01-15T14:35:06.914403+0000","last_fullsized":"2024-01-15T14:35:06.914403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914366+0000","last_change":"2024-01-15T14:35:05.239426+0000","last_active":"2024-01-15T14:35:06.914366+0000","last_peered":"2024-01-15T14:35:06.914366+0000","last_clean":"2024-01-15T14:35:06.914366+0000","last_became_active":"2024-01-15T14:35:05.239139+0000","last_became_peered":"2024-01-15T14:35:05.239139+0000","last_unstale":"2024-01-15T14:35:06.914366+0000","last_undegraded":"2024-01-15T14:35:06.914366+0000","last_fullsized":"2024-01-15T14:35:06.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338008+0000","last_change":"2024-01-15T14:35:04.207140+0000","last_active":"2024-01-15T14:35:06.338008+0000","last_peered":"2024-01-15T14:35:06.338008+0000","last_clean":"2024-01-15T14:35:06.338008+0000","last_became_active":"2024-01-15T14:35:04.206804+0000","last_became_peered":"2024-01-15T14:35:04.206804+0000","last_unstale":"2024-01-15T14:35:06.338008+0000","last_undegraded":"2024-01-15T14:35:06.338008+0000","last_fullsized":"2024-01-15T14:35:06.338008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:30.291286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222686+0000","last_change":"2024-01-15T14:35:05.240267+0000","last_active":"2024-01-15T14:35:06.222686+0000","last_peered":"2024-01-15T14:35:06.222686+0000","last_clean":"2024-01-15T14:35:06.222686+0000","last_became_active":"2024-01-15T14:35:05.239993+0000","last_became_peered":"2024-01-15T14:35:05.239993+0000","last_unstale":"2024-01-15T14:35:06.222686+0000","last_undegraded":"2024-01-15T14:35:06.222686+0000","last_fullsized":"2024-01-15T14:35:06.222686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221632+0000","last_change":"2024-01-15T14:35:04.214296+0000","last_active":"2024-01-15T14:35:06.221632+0000","last_peered":"2024-01-15T14:35:06.221632+0000","last_clean":"2024-01-15T14:35:06.221632+0000","last_became_active":"2024-01-15T14:35:04.213992+0000","last_became_peered":"2024-01-15T14:35:04.213992+0000","last_unstale":"2024-01-15T14:35:06.221632+0000","last_undegraded":"2024-01-15T14:35:06.221632+0000","last_fullsized":"2024-01-15T14:35:06.221632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221677+0000","last_change":"2024-01-15T14:35:05.231106+0000","last_active":"2024-01-15T14:35:06.221677+0000","last_peered":"2024-01-15T14:35:06.221677+0000","last_clean":"2024-01-15T14:35:06.221677+0000","last_became_active":"2024-01-15T14:35:05.230594+0000","last_became_peered":"2024-01-15T14:35:05.230594+0000","last_unstale":"2024-01-15T14:35:06.221677+0000","last_undegraded":"2024-01-15T14:35:06.221677+0000","last_fullsized":"2024-01-15T14:35:06.221677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420056+0000","last_change":"2024-01-15T14:35:04.211246+0000","last_active":"2024-01-15T14:35:06.420056+0000","last_peered":"2024-01-15T14:35:06.420056+0000","last_clean":"2024-01-15T14:35:06.420056+0000","last_became_active":"2024-01-15T14:35:04.210971+0000","last_became_peered":"2024-01-15T14:35:04.210971+0000","last_unstale":"2024-01-15T14:35:06.420056+0000","last_undegraded":"2024-01-15T14:35:06.420056+0000","last_fullsized":"2024-01-15T14:35:06.420056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:06.847713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222962+0000","last_change":"2024-01-15T14:35:05.240373+0000","last_active":"2024-01-15T14:35:06.222962+0000","last_peered":"2024-01-15T14:35:06.222962+0000","last_clean":"2024-01-15T14:35:06.222962+0000","last_became_active":"2024-01-15T14:35:05.240078+0000","last_became_peered":"2024-01-15T14:35:05.240078+0000","last_unstale":"2024-01-15T14:35:06.222962+0000","last_undegraded":"2024-01-15T14:35:06.222962+0000","last_fullsized":"2024-01-15T14:35:06.222962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038437+0000","last_change":"2024-01-15T14:35:04.204121+0000","last_active":"2024-01-15T14:35:07.038437+0000","last_peered":"2024-01-15T14:35:07.038437+0000","last_clean":"2024-01-15T14:35:07.038437+0000","last_became_active":"2024-01-15T14:35:04.203811+0000","last_became_peered":"2024-01-15T14:35:04.203811+0000","last_unstale":"2024-01-15T14:35:07.038437+0000","last_undegraded":"2024-01-15T14:35:07.038437+0000","last_fullsized":"2024-01-15T14:35:07.038437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222934+0000","last_change":"2024-01-15T14:35:05.239960+0000","last_active":"2024-01-15T14:35:06.222934+0000","last_peered":"2024-01-15T14:35:06.222934+0000","last_clean":"2024-01-15T14:35:06.222934+0000","last_became_active":"2024-01-15T14:35:05.239645+0000","last_became_peered":"2024-01-15T14:35:05.239645+0000","last_unstale":"2024-01-15T14:35:06.222934+0000","last_undegraded":"2024-01-15T14:35:06.222934+0000","last_fullsized":"2024-01-15T14:35:06.222934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038563+0000","last_change":"2024-01-15T14:35:04.212055+0000","last_active":"2024-01-15T14:35:07.038563+0000","last_peered":"2024-01-15T14:35:07.038563+0000","last_clean":"2024-01-15T14:35:07.038563+0000","last_became_active":"2024-01-15T14:35:04.211788+0000","last_became_peered":"2024-01-15T14:35:04.211788+0000","last_unstale":"2024-01-15T14:35:07.038563+0000","last_undegraded":"2024-01-15T14:35:07.038563+0000","last_fullsized":"2024-01-15T14:35:07.038563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338047+0000","last_change":"2024-01-15T14:35:05.220108+0000","last_active":"2024-01-15T14:35:06.338047+0000","last_peered":"2024-01-15T14:35:06.338047+0000","last_clean":"2024-01-15T14:35:06.338047+0000","last_became_active":"2024-01-15T14:35:05.219812+0000","last_became_peered":"2024-01-15T14:35:05.219812+0000","last_unstale":"2024-01-15T14:35:06.338047+0000","last_undegraded":"2024-01-15T14:35:06.338047+0000","last_fullsized":"2024-01-15T14:35:06.338047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337937+0000","last_change":"2024-01-15T14:35:04.212432+0000","last_active":"2024-01-15T14:35:06.337937+0000","last_peered":"2024-01-15T14:35:06.337937+0000","last_clean":"2024-01-15T14:35:06.337937+0000","last_became_active":"2024-01-15T14:35:04.211351+0000","last_became_peered":"2024-01-15T14:35:04.211351+0000","last_unstale":"2024-01-15T14:35:06.337937+0000","last_undegraded":"2024-01-15T14:35:06.337937+0000","last_fullsized":"2024-01-15T14:35:06.337937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:38.895254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038566+0000","last_change":"2024-01-15T14:35:05.234938+0000","last_active":"2024-01-15T14:35:07.038566+0000","last_peered":"2024-01-15T14:35:07.038566+0000","last_clean":"2024-01-15T14:35:07.038566+0000","last_became_active":"2024-01-15T14:35:05.234597+0000","last_became_peered":"2024-01-15T14:35:05.234597+0000","last_unstale":"2024-01-15T14:35:07.038566+0000","last_undegraded":"2024-01-15T14:35:07.038566+0000","last_fullsized":"2024-01-15T14:35:07.038566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:00.256458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038392+0000","last_change":"2024-01-15T14:35:05.211465+0000","last_active":"2024-01-15T14:35:07.038392+0000","last_peered":"2024-01-15T14:35:07.038392+0000","last_clean":"2024-01-15T14:35:07.038392+0000","last_became_active":"2024-01-15T14:35:05.211188+0000","last_became_peered":"2024-01-15T14:35:05.211188+0000","last_unstale":"2024-01-15T14:35:07.038392+0000","last_undegraded":"2024-01-15T14:35:07.038392+0000","last_fullsized":"2024-01-15T14:35:07.038392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.419968+0000","last_change":"2024-01-15T14:35:04.208525+0000","last_active":"2024-01-15T14:35:06.419968+0000","last_peered":"2024-01-15T14:35:06.419968+0000","last_clean":"2024-01-15T14:35:06.419968+0000","last_became_active":"2024-01-15T14:35:04.208217+0000","last_became_peered":"2024-01-15T14:35:04.208217+0000","last_unstale":"2024-01-15T14:35:06.419968+0000","last_undegraded":"2024-01-15T14:35:06.419968+0000","last_fullsized":"2024-01-15T14:35:06.419968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220595+0000","last_change":"2024-01-15T14:35:05.236274+0000","last_active":"2024-01-15T14:35:06.220595+0000","last_peered":"2024-01-15T14:35:06.220595+0000","last_clean":"2024-01-15T14:35:06.220595+0000","last_became_active":"2024-01-15T14:35:05.236000+0000","last_became_peered":"2024-01-15T14:35:05.236000+0000","last_unstale":"2024-01-15T14:35:06.220595+0000","last_undegraded":"2024-01-15T14:35:06.220595+0000","last_fullsized":"2024-01-15T14:35:06.220595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337639+0000","last_change":"2024-01-15T14:35:04.209747+0000","last_active":"2024-01-15T14:35:06.337639+0000","last_peered":"2024-01-15T14:35:06.337639+0000","last_clean":"2024-01-15T14:35:06.337639+0000","last_became_active":"2024-01-15T14:35:04.209450+0000","last_became_peered":"2024-01-15T14:35:04.209450+0000","last_unstale":"2024-01-15T14:35:06.337639+0000","last_undegraded":"2024-01-15T14:35:06.337639+0000","last_fullsized":"2024-01-15T14:35:06.337639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:37:47.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T14:35:05.216923+0000","last_change":"2024-01-15T14:34:54.738502+0000","last_active":"2024-01-15T14:35:05.216923+0000","last_peered":"2024-01-15T14:35:05.216923+0000","last_clean":"2024-01-15T14:35:05.216923+0000","last_became_active":"2024-01-15T14:34:54.737354+0000","last_became_peered":"2024-01-15T14:34:54.737354+0000","last_unstale":"2024-01-15T14:35:05.216923+0000","last_undegraded":"2024-01-15T14:35:05.216923+0000","last_fullsized":"2024-01-15T14:35:05.216923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_clean_scrub_stamp":"2024-01-15T14:34:53.124419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:09:14.038173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914075+0000","last_change":"2024-01-15T14:35:05.231021+0000","last_active":"2024-01-15T14:35:06.914075+0000","last_peered":"2024-01-15T14:35:06.914075+0000","last_clean":"2024-01-15T14:35:06.914075+0000","last_became_active":"2024-01-15T14:35:05.230721+0000","last_became_peered":"2024-01-15T14:35:05.230721+0000","last_unstale":"2024-01-15T14:35:06.914075+0000","last_undegraded":"2024-01-15T14:35:06.914075+0000","last_fullsized":"2024-01-15T14:35:06.914075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:15.565484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222435+0000","last_change":"2024-01-15T14:35:04.216271+0000","last_active":"2024-01-15T14:35:06.222435+0000","last_peered":"2024-01-15T14:35:06.222435+0000","last_clean":"2024-01-15T14:35:06.222435+0000","last_became_active":"2024-01-15T14:35:04.215955+0000","last_became_peered":"2024-01-15T14:35:04.215955+0000","last_unstale":"2024-01-15T14:35:06.222435+0000","last_undegraded":"2024-01-15T14:35:06.222435+0000","last_fullsized":"2024-01-15T14:35:06.222435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223126+0000","last_change":"2024-01-15T14:35:04.218008+0000","last_active":"2024-01-15T14:35:06.223126+0000","last_peered":"2024-01-15T14:35:06.223126+0000","last_clean":"2024-01-15T14:35:06.223126+0000","last_became_active":"2024-01-15T14:35:04.217672+0000","last_became_peered":"2024-01-15T14:35:04.217672+0000","last_unstale":"2024-01-15T14:35:06.223126+0000","last_undegraded":"2024-01-15T14:35:06.223126+0000","last_fullsized":"2024-01-15T14:35:06.223126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:25.883807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038622+0000","last_change":"2024-01-15T14:35:05.213577+0000","last_active":"2024-01-15T14:35:07.038622+0000","last_peered":"2024-01-15T14:35:07.038622+0000","last_clean":"2024-01-15T14:35:07.038622+0000","last_became_active":"2024-01-15T14:35:05.212565+0000","last_became_peered":"2024-01-15T14:35:05.212565+0000","last_unstale":"2024-01-15T14:35:07.038622+0000","last_undegraded":"2024-01-15T14:35:07.038622+0000","last_fullsized":"2024-01-15T14:35:07.038622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038646+0000","last_change":"2024-01-15T14:35:04.216515+0000","last_active":"2024-01-15T14:35:07.038646+0000","last_peered":"2024-01-15T14:35:07.038646+0000","last_clean":"2024-01-15T14:35:07.038646+0000","last_became_active":"2024-01-15T14:35:04.216291+0000","last_became_peered":"2024-01-15T14:35:04.216291+0000","last_unstale":"2024-01-15T14:35:07.038646+0000","last_undegraded":"2024-01-15T14:35:07.038646+0000","last_fullsized":"2024-01-15T14:35:07.038646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221002+0000","last_change":"2024-01-15T14:35:05.233144+0000","last_active":"2024-01-15T14:35:06.221002+0000","last_peered":"2024-01-15T14:35:06.221002+0000","last_clean":"2024-01-15T14:35:06.221002+0000","last_became_active":"2024-01-15T14:35:05.232813+0000","last_became_peered":"2024-01-15T14:35:05.232813+0000","last_unstale":"2024-01-15T14:35:06.221002+0000","last_undegraded":"2024-01-15T14:35:06.221002+0000","last_fullsized":"2024-01-15T14:35:06.221002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222447+0000","last_change":"2024-01-15T14:35:04.215265+0000","last_active":"2024-01-15T14:35:06.222447+0000","last_peered":"2024-01-15T14:35:06.222447+0000","last_clean":"2024-01-15T14:35:06.222447+0000","last_became_active":"2024-01-15T14:35:04.214951+0000","last_became_peered":"2024-01-15T14:35:04.214951+0000","last_unstale":"2024-01-15T14:35:06.222447+0000","last_undegraded":"2024-01-15T14:35:06.222447+0000","last_fullsized":"2024-01-15T14:35:06.222447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914377+0000","last_change":"2024-01-15T14:35:05.228381+0000","last_active":"2024-01-15T14:35:06.914377+0000","last_peered":"2024-01-15T14:35:06.914377+0000","last_clean":"2024-01-15T14:35:06.914377+0000","last_became_active":"2024-01-15T14:35:05.228101+0000","last_became_peered":"2024-01-15T14:35:05.228101+0000","last_unstale":"2024-01-15T14:35:06.914377+0000","last_undegraded":"2024-01-15T14:35:06.914377+0000","last_fullsized":"2024-01-15T14:35:06.914377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222345+0000","last_change":"2024-01-15T14:35:04.215917+0000","last_active":"2024-01-15T14:35:06.222345+0000","last_peered":"2024-01-15T14:35:06.222345+0000","last_clean":"2024-01-15T14:35:06.222345+0000","last_became_active":"2024-01-15T14:35:04.215592+0000","last_became_peered":"2024-01-15T14:35:04.215592+0000","last_unstale":"2024-01-15T14:35:06.222345+0000","last_undegraded":"2024-01-15T14:35:06.222345+0000","last_fullsized":"2024-01-15T14:35:06.222345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038319+0000","last_change":"2024-01-15T14:35:05.233251+0000","last_active":"2024-01-15T14:35:07.038319+0000","last_peered":"2024-01-15T14:35:07.038319+0000","last_clean":"2024-01-15T14:35:07.038319+0000","last_became_active":"2024-01-15T14:35:05.232982+0000","last_became_peered":"2024-01-15T14:35:05.232982+0000","last_unstale":"2024-01-15T14:35:07.038319+0000","last_undegraded":"2024-01-15T14:35:07.038319+0000","last_fullsized":"2024-01-15T14:35:07.038319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:15.827108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038435+0000","last_change":"2024-01-15T14:35:04.218539+0000","last_active":"2024-01-15T14:35:07.038435+0000","last_peered":"2024-01-15T14:35:07.038435+0000","last_clean":"2024-01-15T14:35:07.038435+0000","last_became_active":"2024-01-15T14:35:04.218307+0000","last_became_peered":"2024-01-15T14:35:04.218307+0000","last_unstale":"2024-01-15T14:35:07.038435+0000","last_undegraded":"2024-01-15T14:35:07.038435+0000","last_fullsized":"2024-01-15T14:35:07.038435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220625+0000","last_change":"2024-01-15T14:35:05.220403+0000","last_active":"2024-01-15T14:35:06.220625+0000","last_peered":"2024-01-15T14:35:06.220625+0000","last_clean":"2024-01-15T14:35:06.220625+0000","last_became_active":"2024-01-15T14:35:05.220033+0000","last_became_peered":"2024-01-15T14:35:05.220033+0000","last_unstale":"2024-01-15T14:35:06.220625+0000","last_undegraded":"2024-01-15T14:35:06.220625+0000","last_fullsized":"2024-01-15T14:35:06.220625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038582+0000","last_change":"2024-01-15T14:35:04.211061+0000","last_active":"2024-01-15T14:35:07.038582+0000","last_peered":"2024-01-15T14:35:07.038582+0000","last_clean":"2024-01-15T14:35:07.038582+0000","last_became_active":"2024-01-15T14:35:04.210748+0000","last_became_peered":"2024-01-15T14:35:04.210748+0000","last_unstale":"2024-01-15T14:35:07.038582+0000","last_undegraded":"2024-01-15T14:35:07.038582+0000","last_fullsized":"2024-01-15T14:35:07.038582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220819+0000","last_change":"2024-01-15T14:35:05.232247+0000","last_active":"2024-01-15T14:35:06.220819+0000","last_peered":"2024-01-15T14:35:06.220819+0000","last_clean":"2024-01-15T14:35:06.220819+0000","last_became_active":"2024-01-15T14:35:05.231914+0000","last_became_peered":"2024-01-15T14:35:05.231914+0000","last_unstale":"2024-01-15T14:35:06.220819+0000","last_undegraded":"2024-01-15T14:35:06.220819+0000","last_fullsized":"2024-01-15T14:35:06.220819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222720+0000","last_change":"2024-01-15T14:35:04.216586+0000","last_active":"2024-01-15T14:35:06.222720+0000","last_peered":"2024-01-15T14:35:06.222720+0000","last_clean":"2024-01-15T14:35:06.222720+0000","last_became_active":"2024-01-15T14:35:04.216231+0000","last_became_peered":"2024-01-15T14:35:04.216231+0000","last_unstale":"2024-01-15T14:35:06.222720+0000","last_undegraded":"2024-01-15T14:35:06.222720+0000","last_fullsized":"2024-01-15T14:35:06.222720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221812+0000","last_change":"2024-01-15T14:35:05.237033+0000","last_active":"2024-01-15T14:35:06.221812+0000","last_peered":"2024-01-15T14:35:06.221812+0000","last_clean":"2024-01-15T14:35:06.221812+0000","last_became_active":"2024-01-15T14:35:05.236738+0000","last_became_peered":"2024-01-15T14:35:05.236738+0000","last_unstale":"2024-01-15T14:35:06.221812+0000","last_undegraded":"2024-01-15T14:35:06.221812+0000","last_fullsized":"2024-01-15T14:35:06.221812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420017+0000","last_change":"2024-01-15T14:35:04.208500+0000","last_active":"2024-01-15T14:35:06.420017+0000","last_peered":"2024-01-15T14:35:06.420017+0000","last_clean":"2024-01-15T14:35:06.420017+0000","last_became_active":"2024-01-15T14:35:04.208227+0000","last_became_peered":"2024-01-15T14:35:04.208227+0000","last_unstale":"2024-01-15T14:35:06.420017+0000","last_undegraded":"2024-01-15T14:35:06.420017+0000","last_fullsized":"2024-01-15T14:35:06.420017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:11:11.486899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223119+0000","last_change":"2024-01-15T14:35:05.232373+0000","last_active":"2024-01-15T14:35:06.223119+0000","last_peered":"2024-01-15T14:35:06.223119+0000","last_clean":"2024-01-15T14:35:06.223119+0000","last_became_active":"2024-01-15T14:35:05.232098+0000","last_became_peered":"2024-01-15T14:35:05.232098+0000","last_unstale":"2024-01-15T14:35:06.223119+0000","last_undegraded":"2024-01-15T14:35:06.223119+0000","last_fullsized":"2024-01-15T14:35:06.223119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038754+0000","last_change":"2024-01-15T14:35:04.213296+0000","last_active":"2024-01-15T14:35:07.038754+0000","last_peered":"2024-01-15T14:35:07.038754+0000","last_clean":"2024-01-15T14:35:07.038754+0000","last_became_active":"2024-01-15T14:35:04.212955+0000","last_became_peered":"2024-01-15T14:35:04.212955+0000","last_unstale":"2024-01-15T14:35:07.038754+0000","last_undegraded":"2024-01-15T14:35:07.038754+0000","last_fullsized":"2024-01-15T14:35:07.038754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:10.265464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338036+0000","last_change":"2024-01-15T14:35:05.219979+0000","last_active":"2024-01-15T14:35:06.338036+0000","last_peered":"2024-01-15T14:35:06.338036+0000","last_clean":"2024-01-15T14:35:06.338036+0000","last_became_active":"2024-01-15T14:35:05.219668+0000","last_became_peered":"2024-01-15T14:35:05.219668+0000","last_unstale":"2024-01-15T14:35:06.338036+0000","last_undegraded":"2024-01-15T14:35:06.338036+0000","last_fullsized":"2024-01-15T14:35:06.338036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420118+0000","last_change":"2024-01-15T14:35:04.208970+0000","last_active":"2024-01-15T14:35:06.420118+0000","last_peered":"2024-01-15T14:35:06.420118+0000","last_clean":"2024-01-15T14:35:06.420118+0000","last_became_active":"2024-01-15T14:35:04.208699+0000","last_became_peered":"2024-01-15T14:35:04.208699+0000","last_unstale":"2024-01-15T14:35:06.420118+0000","last_undegraded":"2024-01-15T14:35:06.420118+0000","last_fullsized":"2024-01-15T14:35:06.420118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223420+0000","last_change":"2024-01-15T14:35:05.239873+0000","last_active":"2024-01-15T14:35:06.223420+0000","last_peered":"2024-01-15T14:35:06.223420+0000","last_clean":"2024-01-15T14:35:06.223420+0000","last_became_active":"2024-01-15T14:35:05.239571+0000","last_became_peered":"2024-01-15T14:35:05.239571+0000","last_unstale":"2024-01-15T14:35:06.223420+0000","last_undegraded":"2024-01-15T14:35:06.223420+0000","last_fullsized":"2024-01-15T14:35:06.223420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221810+0000","last_change":"2024-01-15T14:35:05.230887+0000","last_active":"2024-01-15T14:35:06.221810+0000","last_peered":"2024-01-15T14:35:06.221810+0000","last_clean":"2024-01-15T14:35:06.221810+0000","last_became_active":"2024-01-15T14:35:05.230187+0000","last_became_peered":"2024-01-15T14:35:05.230187+0000","last_unstale":"2024-01-15T14:35:06.221810+0000","last_undegraded":"2024-01-15T14:35:06.221810+0000","last_fullsized":"2024-01-15T14:35:06.221810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420110+0000","last_change":"2024-01-15T14:35:05.217285+0000","last_active":"2024-01-15T14:35:06.420110+0000","last_peered":"2024-01-15T14:35:06.420110+0000","last_clean":"2024-01-15T14:35:06.420110+0000","last_became_active":"2024-01-15T14:35:05.217015+0000","last_became_peered":"2024-01-15T14:35:05.217015+0000","last_unstale":"2024-01-15T14:35:06.420110+0000","last_undegraded":"2024-01-15T14:35:06.420110+0000","last_fullsized":"2024-01-15T14:35:06.420110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337918+0000","last_change":"2024-01-15T14:35:05.220986+0000","last_active":"2024-01-15T14:35:06.337918+0000","last_peered":"2024-01-15T14:35:06.337918+0000","last_clean":"2024-01-15T14:35:06.337918+0000","last_became_active":"2024-01-15T14:35:05.220704+0000","last_became_peered":"2024-01-15T14:35:05.220704+0000","last_unstale":"2024-01-15T14:35:06.337918+0000","last_undegraded":"2024-01-15T14:35:06.337918+0000","last_fullsized":"2024-01-15T14:35:06.337918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914367+0000","last_change":"2024-01-15T14:35:05.232048+0000","last_active":"2024-01-15T14:35:06.914367+0000","last_peered":"2024-01-15T14:35:06.914367+0000","last_clean":"2024-01-15T14:35:06.914367+0000","last_became_active":"2024-01-15T14:35:05.231708+0000","last_became_peered":"2024-01-15T14:35:05.231708+0000","last_unstale":"2024-01-15T14:35:06.914367+0000","last_undegraded":"2024-01-15T14:35:06.914367+0000","last_fullsized":"2024-01-15T14:35:06.914367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220904+0000","last_change":"2024-01-15T14:35:05.236304+0000","last_active":"2024-01-15T14:35:06.220904+0000","last_peered":"2024-01-15T14:35:06.220904+0000","last_clean":"2024-01-15T14:35:06.220904+0000","last_became_active":"2024-01-15T14:35:05.235435+0000","last_became_peered":"2024-01-15T14:35:05.235435+0000","last_unstale":"2024-01-15T14:35:06.220904+0000","last_undegraded":"2024-01-15T14:35:06.220904+0000","last_fullsized":"2024-01-15T14:35:06.220904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038669+0000","last_change":"2024-01-15T14:35:05.216092+0000","last_active":"2024-01-15T14:35:07.038669+0000","last_peered":"2024-01-15T14:35:07.038669+0000","last_clean":"2024-01-15T14:35:07.038669+0000","last_became_active":"2024-01-15T14:35:05.215618+0000","last_became_peered":"2024-01-15T14:35:05.215618+0000","last_unstale":"2024-01-15T14:35:07.038669+0000","last_undegraded":"2024-01-15T14:35:07.038669+0000","last_fullsized":"2024-01-15T14:35:07.038669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222044+0000","last_change":"2024-01-15T14:35:05.229045+0000","last_active":"2024-01-15T14:35:06.222044+0000","last_peered":"2024-01-15T14:35:06.222044+0000","last_clean":"2024-01-15T14:35:06.222044+0000","last_became_active":"2024-01-15T14:35:05.228771+0000","last_became_peered":"2024-01-15T14:35:05.228771+0000","last_unstale":"2024-01-15T14:35:06.222044+0000","last_undegraded":"2024-01-15T14:35:06.222044+0000","last_fullsized":"2024-01-15T14:35:06.222044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220972+0000","last_change":"2024-01-15T14:35:05.235711+0000","last_active":"2024-01-15T14:35:06.220972+0000","last_peered":"2024-01-15T14:35:06.220972+0000","last_clean":"2024-01-15T14:35:06.220972+0000","last_became_active":"2024-01-15T14:35:05.235419+0000","last_became_peered":"2024-01-15T14:35:05.235419+0000","last_unstale":"2024-01-15T14:35:06.220972+0000","last_undegraded":"2024-01-15T14:35:06.220972+0000","last_fullsized":"2024-01-15T14:35:06.220972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337825+0000","last_change":"2024-01-15T14:35:05.221416+0000","last_active":"2024-01-15T14:35:06.337825+0000","last_peered":"2024-01-15T14:35:06.337825+0000","last_clean":"2024-01-15T14:35:06.337825+0000","last_became_active":"2024-01-15T14:35:05.221124+0000","last_became_peered":"2024-01-15T14:35:05.221124+0000","last_unstale":"2024-01-15T14:35:06.337825+0000","last_undegraded":"2024-01-15T14:35:06.337825+0000","last_fullsized":"2024-01-15T14:35:06.337825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220561+0000","last_change":"2024-01-15T14:35:05.235785+0000","last_active":"2024-01-15T14:35:06.220561+0000","last_peered":"2024-01-15T14:35:06.220561+0000","last_clean":"2024-01-15T14:35:06.220561+0000","last_became_active":"2024-01-15T14:35:05.235193+0000","last_became_peered":"2024-01-15T14:35:05.235193+0000","last_unstale":"2024-01-15T14:35:06.220561+0000","last_undegraded":"2024-01-15T14:35:06.220561+0000","last_fullsized":"2024-01-15T14:35:06.220561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337729+0000","last_change":"2024-01-15T14:35:05.225944+0000","last_active":"2024-01-15T14:35:06.337729+0000","last_peered":"2024-01-15T14:35:06.337729+0000","last_clean":"2024-01-15T14:35:06.337729+0000","last_became_active":"2024-01-15T14:35:05.225660+0000","last_became_peered":"2024-01-15T14:35:05.225660+0000","last_unstale":"2024-01-15T14:35:06.337729+0000","last_undegraded":"2024-01-15T14:35:06.337729+0000","last_fullsized":"2024-01-15T14:35:06.337729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914286+0000","last_change":"2024-01-15T14:35:05.243644+0000","last_active":"2024-01-15T14:35:06.914286+0000","last_peered":"2024-01-15T14:35:06.914286+0000","last_clean":"2024-01-15T14:35:06.914286+0000","last_became_active":"2024-01-15T14:35:05.243362+0000","last_became_peered":"2024-01-15T14:35:05.243362+0000","last_unstale":"2024-01-15T14:35:06.914286+0000","last_undegraded":"2024-01-15T14:35:06.914286+0000","last_fullsized":"2024-01-15T14:35:06.914286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:58.010415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222177+0000","last_change":"2024-01-15T14:35:05.237833+0000","last_active":"2024-01-15T14:35:06.222177+0000","last_peered":"2024-01-15T14:35:06.222177+0000","last_clean":"2024-01-15T14:35:06.222177+0000","last_became_active":"2024-01-15T14:35:05.237576+0000","last_became_peered":"2024-01-15T14:35:05.237576+0000","last_unstale":"2024-01-15T14:35:06.222177+0000","last_undegraded":"2024-01-15T14:35:06.222177+0000","last_fullsized":"2024-01-15T14:35:06.222177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T14:35:09.316 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T14:35:09.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.472+0000 7f34ec98e700 1 Processor -- start 2024-01-15T14:35:09.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.472+0000 7f34ec98e700 1 -- start start 2024-01-15T14:35:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34ec98e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e815bed0 0x7f34e80b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34ec98e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e81603c0 0x7f34e80d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34ec98e700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80d0770 con 0x7f34e80d1a50 2024-01-15T14:35:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34ec98e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e815c2b0 con 0x7f34e815bed0 2024-01-15T14:35:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34ec98e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80cd100 con 0x7f34e81603c0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e815bed0 0x7f34e80b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e815bed0 0x7f34e80b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57296/0 (socket says 172.21.15.77:57296) 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 -- 172.21.15.77:0/4231797196 learned_addr learned my addr 172.21.15.77:0/4231797196 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e5d9b700 1 --2- 172.21.15.77:0/4231797196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e81603c0 0x7f34e80d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 -- 172.21.15.77:0/4231797196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e81603c0 msgr2=0x7f34e80d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 --2- 172.21.15.77:0/4231797196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e81603c0 0x7f34e80d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 -- 172.21.15.77:0/4231797196 >> v1:172.21.15.159:6789/0 conn(0x7f34e80d1a50 legacy=0x7f34e80d1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.473+0000 7f34e659c700 1 -- 172.21.15.77:0/4231797196 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80d09a0 con 0x7f34e815bed0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34e659c700 1 --2- 172.21.15.77:0/4231797196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e815bed0 0x7f34e80b19f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0098f0 tx=0x7f34dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4c84955ff89c7c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34e559a700 1 -- 172.21.15.77:0/4231797196 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc011830 con 0x7f34e815bed0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34e559a700 1 -- 172.21.15.77:0/4231797196 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc011990 con 0x7f34e815bed0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34e559a700 1 -- 172.21.15.77:0/4231797196 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc011d20 con 0x7f34e815bed0 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 -- 172.21.15.77:0/4231797196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e815bed0 msgr2=0x7f34e80b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 --2- 172.21.15.77:0/4231797196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e815bed0 0x7f34e80b19f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0098f0 tx=0x7f34dc009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 -- 172.21.15.77:0/4231797196 shutdown_connections 2024-01-15T14:35:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 --2- 172.21.15.77:0/4231797196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e815bed0 0x7f34e80b19f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 --2- 172.21.15.77:0/4231797196 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e81603c0 0x7f34e80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 -- 172.21.15.77:0/4231797196 >> 172.21.15.77:0/4231797196 conn(0x7f34e8073130 msgr2=0x7f34e809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 -- 172.21.15.77:0/4231797196 shutdown_connections 2024-01-15T14:35:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.474+0000 7f34ec98e700 1 -- 172.21.15.77:0/4231797196 wait complete. 2024-01-15T14:35:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 Processor -- start 2024-01-15T14:35:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 -- start start 2024-01-15T14:35:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e80d1a50 0x7f34e815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34e815bed0 0x7f34e80c4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 0x7f34e80c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e815dfb0 con 0x7f34e81603c0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e815e290 con 0x7f34e815bed0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.476+0000 7f34ec98e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e815e570 con 0x7f34e80d1a50 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 0x7f34e80c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 0x7f34e80c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57308/0 (socket says 172.21.15.77:57308) 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 -- 172.21.15.77:0/3656682764 learned_addr learned my addr 172.21.15.77:0/3656682764 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e5d9b700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34e815bed0 0x7f34e80c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e659c700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e80d1a50 0x7f34e815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 -- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e80d1a50 msgr2=0x7f34e815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e80d1a50 0x7f34e815d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 -- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34e815bed0 msgr2=0x7f34e80c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34e815bed0 0x7f34e80c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 -- 172.21.15.77:0/3656682764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc009550 con 0x7f34e81603c0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e659c700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e80d1a50 0x7f34e815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34e6d9d700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 0x7f34e80c6d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f34e000cbc0 tx=0x7f34e000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0003c00 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0003d60 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e815e850 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e000ab40 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.477+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e815dcc0 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.478+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e80cca90 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.478+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f34e000aca0 con 0x7f34e81603c0 2024-01-15T14:35:09.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.479+0000 7f34d6ffd700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f34c806b350 0x7f34c806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.479+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34e0089bf0 con 0x7f34e81603c0 2024-01-15T14:35:09.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.479+0000 7f34e659c700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f34c806b350 0x7f34c806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.479+0000 7f34e659c700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f34c806b350 0x7f34c806d810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009c40 tx=0x7f34dc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.484+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34e80cca90 con 0x7f34e81603c0 2024-01-15T14:35:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.670+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34e80ccd90 con 0x7f34c806b350 2024-01-15T14:35:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.677+0000 7f34d6ffd700 1 -- 172.21.15.77:0/3656682764 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264671 (secure 0 0 0) 0x7f34dc011c80 con 0x7f34c806b350 2024-01-15T14:35:09.681 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f34c806b350 msgr2=0x7f34c806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f34c806b350 0x7f34c806d810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009c40 tx=0x7f34dc008030 comp rx=0 tx=0).stop 2024-01-15T14:35:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 msgr2=0x7f34e80c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 0x7f34e80c6d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f34e000cbc0 tx=0x7f34e000ced0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 shutdown_connections 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f34c806b350 0x7f34c806d810 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f34e80d1a50 0x7f34e815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e81603c0 0x7f34e80c6d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 --2- 172.21.15.77:0/3656682764 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f34e815bed0 0x7f34e80c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 >> 172.21.15.77:0/3656682764 conn(0x7f34e8073130 msgr2=0x7f34e80cef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 shutdown_connections 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.680+0000 7f34ec98e700 1 -- 172.21.15.77:0/3656682764 wait complete. 2024-01-15T14:35:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-01-15T14:35:09.692 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-15T14:35:09.018704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219056,"kb_used_data":2860,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755664,"statfs":{"total":773094113280,"available":772869799936,"internally_reserved":0,"allocated":2928640,"data_stored":1758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.832059"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223558+0000","last_change":"2024-01-15T14:35:05.242883+0000","last_active":"2024-01-15T14:35:06.223558+0000","last_peered":"2024-01-15T14:35:06.223558+0000","last_clean":"2024-01-15T14:35:06.223558+0000","last_became_active":"2024-01-15T14:35:05.242650+0000","last_became_peered":"2024-01-15T14:35:05.242650+0000","last_unstale":"2024-01-15T14:35:06.223558+0000","last_undegraded":"2024-01-15T14:35:06.223558+0000","last_fullsized":"2024-01-15T14:35:06.223558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:06:13.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038751+0000","last_change":"2024-01-15T14:35:05.230934+0000","last_active":"2024-01-15T14:35:07.038751+0000","last_peered":"2024-01-15T14:35:07.038751+0000","last_clean":"2024-01-15T14:35:07.038751+0000","last_became_active":"2024-01-15T14:35:05.230356+0000","last_became_peered":"2024-01-15T14:35:05.230356+0000","last_unstale":"2024-01-15T14:35:07.038751+0000","last_undegraded":"2024-01-15T14:35:07.038751+0000","last_fullsized":"2024-01-15T14:35:07.038751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914251+0000","last_change":"2024-01-15T14:35:05.228307+0000","last_active":"2024-01-15T14:35:06.914251+0000","last_peered":"2024-01-15T14:35:06.914251+0000","last_clean":"2024-01-15T14:35:06.914251+0000","last_became_active":"2024-01-15T14:35:05.227876+0000","last_became_peered":"2024-01-15T14:35:05.227876+0000","last_unstale":"2024-01-15T14:35:06.914251+0000","last_undegraded":"2024-01-15T14:35:06.914251+0000","last_fullsized":"2024-01-15T14:35:06.914251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:11.269770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338166+0000","last_change":"2024-01-15T14:35:05.236667+0000","last_active":"2024-01-15T14:35:06.338166+0000","last_peered":"2024-01-15T14:35:06.338166+0000","last_clean":"2024-01-15T14:35:06.338166+0000","last_became_active":"2024-01-15T14:35:05.236363+0000","last_became_peered":"2024-01-15T14:35:05.236363+0000","last_unstale":"2024-01-15T14:35:06.338166+0000","last_undegraded":"2024-01-15T14:35:06.338166+0000","last_fullsized":"2024-01-15T14:35:06.338166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222610+0000","last_change":"2024-01-15T14:35:05.239721+0000","last_active":"2024-01-15T14:35:06.222610+0000","last_peered":"2024-01-15T14:35:06.222610+0000","last_clean":"2024-01-15T14:35:06.222610+0000","last_became_active":"2024-01-15T14:35:05.235648+0000","last_became_peered":"2024-01-15T14:35:05.235648+0000","last_unstale":"2024-01-15T14:35:06.222610+0000","last_undegraded":"2024-01-15T14:35:06.222610+0000","last_fullsized":"2024-01-15T14:35:06.222610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337634+0000","last_change":"2024-01-15T14:35:05.230388+0000","last_active":"2024-01-15T14:35:06.337634+0000","last_peered":"2024-01-15T14:35:06.337634+0000","last_clean":"2024-01-15T14:35:06.337634+0000","last_became_active":"2024-01-15T14:35:05.230109+0000","last_became_peered":"2024-01-15T14:35:05.230109+0000","last_unstale":"2024-01-15T14:35:06.337634+0000","last_undegraded":"2024-01-15T14:35:06.337634+0000","last_fullsized":"2024-01-15T14:35:06.337634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222611+0000","last_change":"2024-01-15T14:35:05.239457+0000","last_active":"2024-01-15T14:35:06.222611+0000","last_peered":"2024-01-15T14:35:06.222611+0000","last_clean":"2024-01-15T14:35:06.222611+0000","last_became_active":"2024-01-15T14:35:05.239152+0000","last_became_peered":"2024-01-15T14:35:05.239152+0000","last_unstale":"2024-01-15T14:35:06.222611+0000","last_undegraded":"2024-01-15T14:35:06.222611+0000","last_fullsized":"2024-01-15T14:35:06.222611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220500+0000","last_change":"2024-01-15T14:35:05.219905+0000","last_active":"2024-01-15T14:35:06.220500+0000","last_peered":"2024-01-15T14:35:06.220500+0000","last_clean":"2024-01-15T14:35:06.220500+0000","last_became_active":"2024-01-15T14:35:05.219331+0000","last_became_peered":"2024-01-15T14:35:05.219331+0000","last_unstale":"2024-01-15T14:35:06.220500+0000","last_undegraded":"2024-01-15T14:35:06.220500+0000","last_fullsized":"2024-01-15T14:35:06.220500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223409+0000","last_change":"2024-01-15T14:35:05.232665+0000","last_active":"2024-01-15T14:35:06.223409+0000","last_peered":"2024-01-15T14:35:06.223409+0000","last_clean":"2024-01-15T14:35:06.223409+0000","last_became_active":"2024-01-15T14:35:05.232381+0000","last_became_peered":"2024-01-15T14:35:05.232381+0000","last_unstale":"2024-01-15T14:35:06.223409+0000","last_undegraded":"2024-01-15T14:35:06.223409+0000","last_fullsized":"2024-01-15T14:35:06.223409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222267+0000","last_change":"2024-01-15T14:35:05.222078+0000","last_active":"2024-01-15T14:35:06.222267+0000","last_peered":"2024-01-15T14:35:06.222267+0000","last_clean":"2024-01-15T14:35:06.222267+0000","last_became_active":"2024-01-15T14:35:05.221809+0000","last_became_peered":"2024-01-15T14:35:05.221809+0000","last_unstale":"2024-01-15T14:35:06.222267+0000","last_undegraded":"2024-01-15T14:35:06.222267+0000","last_fullsized":"2024-01-15T14:35:06.222267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221969+0000","last_change":"2024-01-15T14:35:05.222838+0000","last_active":"2024-01-15T14:35:06.221969+0000","last_peered":"2024-01-15T14:35:06.221969+0000","last_clean":"2024-01-15T14:35:06.221969+0000","last_became_active":"2024-01-15T14:35:05.221973+0000","last_became_peered":"2024-01-15T14:35:05.221973+0000","last_unstale":"2024-01-15T14:35:06.221969+0000","last_undegraded":"2024-01-15T14:35:06.221969+0000","last_fullsized":"2024-01-15T14:35:06.221969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038638+0000","last_change":"2024-01-15T14:35:05.242599+0000","last_active":"2024-01-15T14:35:07.038638+0000","last_peered":"2024-01-15T14:35:07.038638+0000","last_clean":"2024-01-15T14:35:07.038638+0000","last_became_active":"2024-01-15T14:35:05.242171+0000","last_became_peered":"2024-01-15T14:35:05.242171+0000","last_unstale":"2024-01-15T14:35:07.038638+0000","last_undegraded":"2024-01-15T14:35:07.038638+0000","last_fullsized":"2024-01-15T14:35:07.038638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220701+0000","last_change":"2024-01-15T14:35:05.224493+0000","last_active":"2024-01-15T14:35:06.220701+0000","last_peered":"2024-01-15T14:35:06.220701+0000","last_clean":"2024-01-15T14:35:06.220701+0000","last_became_active":"2024-01-15T14:35:05.224166+0000","last_became_peered":"2024-01-15T14:35:05.224166+0000","last_unstale":"2024-01-15T14:35:06.220701+0000","last_undegraded":"2024-01-15T14:35:06.220701+0000","last_fullsized":"2024-01-15T14:35:06.220701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914368+0000","last_change":"2024-01-15T14:35:05.229113+0000","last_active":"2024-01-15T14:35:06.914368+0000","last_peered":"2024-01-15T14:35:06.914368+0000","last_clean":"2024-01-15T14:35:06.914368+0000","last_became_active":"2024-01-15T14:35:05.228790+0000","last_became_peered":"2024-01-15T14:35:05.228790+0000","last_unstale":"2024-01-15T14:35:06.914368+0000","last_undegraded":"2024-01-15T14:35:06.914368+0000","last_fullsized":"2024-01-15T14:35:06.914368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038480+0000","last_change":"2024-01-15T14:35:05.236987+0000","last_active":"2024-01-15T14:35:07.038480+0000","last_peered":"2024-01-15T14:35:07.038480+0000","last_clean":"2024-01-15T14:35:07.038480+0000","last_became_active":"2024-01-15T14:35:05.236693+0000","last_became_peered":"2024-01-15T14:35:05.236693+0000","last_unstale":"2024-01-15T14:35:07.038480+0000","last_undegraded":"2024-01-15T14:35:07.038480+0000","last_fullsized":"2024-01-15T14:35:07.038480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337512+0000","last_change":"2024-01-15T14:35:05.229665+0000","last_active":"2024-01-15T14:35:06.337512+0000","last_peered":"2024-01-15T14:35:06.337512+0000","last_clean":"2024-01-15T14:35:06.337512+0000","last_became_active":"2024-01-15T14:35:05.229391+0000","last_became_peered":"2024-01-15T14:35:05.229391+0000","last_unstale":"2024-01-15T14:35:06.337512+0000","last_undegraded":"2024-01-15T14:35:06.337512+0000","last_fullsized":"2024-01-15T14:35:06.337512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338230+0000","last_change":"2024-01-15T14:35:05.232892+0000","last_active":"2024-01-15T14:35:06.338230+0000","last_peered":"2024-01-15T14:35:06.338230+0000","last_clean":"2024-01-15T14:35:06.338230+0000","last_became_active":"2024-01-15T14:35:05.232596+0000","last_became_peered":"2024-01-15T14:35:05.232596+0000","last_unstale":"2024-01-15T14:35:06.338230+0000","last_undegraded":"2024-01-15T14:35:06.338230+0000","last_fullsized":"2024-01-15T14:35:06.338230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420132+0000","last_change":"2024-01-15T14:35:05.224599+0000","last_active":"2024-01-15T14:35:06.420132+0000","last_peered":"2024-01-15T14:35:06.420132+0000","last_clean":"2024-01-15T14:35:06.420132+0000","last_became_active":"2024-01-15T14:35:05.224306+0000","last_became_peered":"2024-01-15T14:35:05.224306+0000","last_unstale":"2024-01-15T14:35:06.420132+0000","last_undegraded":"2024-01-15T14:35:06.420132+0000","last_fullsized":"2024-01-15T14:35:06.420132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:35.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222848+0000","last_change":"2024-01-15T14:35:05.235613+0000","last_active":"2024-01-15T14:35:06.222848+0000","last_peered":"2024-01-15T14:35:06.222848+0000","last_clean":"2024-01-15T14:35:06.222848+0000","last_became_active":"2024-01-15T14:35:05.235298+0000","last_became_peered":"2024-01-15T14:35:05.235298+0000","last_unstale":"2024-01-15T14:35:06.222848+0000","last_undegraded":"2024-01-15T14:35:06.222848+0000","last_fullsized":"2024-01-15T14:35:06.222848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914462+0000","last_change":"2024-01-15T14:35:04.217677+0000","last_active":"2024-01-15T14:35:06.914462+0000","last_peered":"2024-01-15T14:35:06.914462+0000","last_clean":"2024-01-15T14:35:06.914462+0000","last_became_active":"2024-01-15T14:35:04.217361+0000","last_became_peered":"2024-01-15T14:35:04.217361+0000","last_unstale":"2024-01-15T14:35:06.914462+0000","last_undegraded":"2024-01-15T14:35:06.914462+0000","last_fullsized":"2024-01-15T14:35:06.914462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220970+0000","last_change":"2024-01-15T14:35:05.225192+0000","last_active":"2024-01-15T14:35:06.220970+0000","last_peered":"2024-01-15T14:35:06.220970+0000","last_clean":"2024-01-15T14:35:06.220970+0000","last_became_active":"2024-01-15T14:35:05.224634+0000","last_became_peered":"2024-01-15T14:35:05.224634+0000","last_unstale":"2024-01-15T14:35:06.220970+0000","last_undegraded":"2024-01-15T14:35:06.220970+0000","last_fullsized":"2024-01-15T14:35:06.220970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220874+0000","last_change":"2024-01-15T14:35:04.204838+0000","last_active":"2024-01-15T14:35:06.220874+0000","last_peered":"2024-01-15T14:35:06.220874+0000","last_clean":"2024-01-15T14:35:06.220874+0000","last_became_active":"2024-01-15T14:35:04.203651+0000","last_became_peered":"2024-01-15T14:35:04.203651+0000","last_unstale":"2024-01-15T14:35:06.220874+0000","last_undegraded":"2024-01-15T14:35:06.220874+0000","last_fullsized":"2024-01-15T14:35:06.220874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914437+0000","last_change":"2024-01-15T14:35:05.228513+0000","last_active":"2024-01-15T14:35:06.914437+0000","last_peered":"2024-01-15T14:35:06.914437+0000","last_clean":"2024-01-15T14:35:06.914437+0000","last_became_active":"2024-01-15T14:35:05.228229+0000","last_became_peered":"2024-01-15T14:35:05.228229+0000","last_unstale":"2024-01-15T14:35:06.914437+0000","last_undegraded":"2024-01-15T14:35:06.914437+0000","last_fullsized":"2024-01-15T14:35:06.914437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223203+0000","last_change":"2024-01-15T14:35:04.217602+0000","last_active":"2024-01-15T14:35:06.223203+0000","last_peered":"2024-01-15T14:35:06.223203+0000","last_clean":"2024-01-15T14:35:06.223203+0000","last_became_active":"2024-01-15T14:35:04.217271+0000","last_became_peered":"2024-01-15T14:35:04.217271+0000","last_unstale":"2024-01-15T14:35:06.223203+0000","last_undegraded":"2024-01-15T14:35:06.223203+0000","last_fullsized":"2024-01-15T14:35:06.223203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:24.748909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337912+0000","last_change":"2024-01-15T14:35:05.234884+0000","last_active":"2024-01-15T14:35:06.337912+0000","last_peered":"2024-01-15T14:35:06.337912+0000","last_clean":"2024-01-15T14:35:06.337912+0000","last_became_active":"2024-01-15T14:35:05.234556+0000","last_became_peered":"2024-01-15T14:35:05.234556+0000","last_unstale":"2024-01-15T14:35:06.337912+0000","last_undegraded":"2024-01-15T14:35:06.337912+0000","last_fullsized":"2024-01-15T14:35:06.337912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222800+0000","last_change":"2024-01-15T14:35:04.217549+0000","last_active":"2024-01-15T14:35:06.222800+0000","last_peered":"2024-01-15T14:35:06.222800+0000","last_clean":"2024-01-15T14:35:06.222800+0000","last_became_active":"2024-01-15T14:35:04.217245+0000","last_became_peered":"2024-01-15T14:35:04.217245+0000","last_unstale":"2024-01-15T14:35:06.222800+0000","last_undegraded":"2024-01-15T14:35:06.222800+0000","last_fullsized":"2024-01-15T14:35:06.222800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337934+0000","last_change":"2024-01-15T14:35:05.230835+0000","last_active":"2024-01-15T14:35:06.337934+0000","last_peered":"2024-01-15T14:35:06.337934+0000","last_clean":"2024-01-15T14:35:06.337934+0000","last_became_active":"2024-01-15T14:35:05.230395+0000","last_became_peered":"2024-01-15T14:35:05.230395+0000","last_unstale":"2024-01-15T14:35:06.337934+0000","last_undegraded":"2024-01-15T14:35:06.337934+0000","last_fullsized":"2024-01-15T14:35:06.337934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220762+0000","last_change":"2024-01-15T14:35:04.216108+0000","last_active":"2024-01-15T14:35:06.220762+0000","last_peered":"2024-01-15T14:35:06.220762+0000","last_clean":"2024-01-15T14:35:06.220762+0000","last_became_active":"2024-01-15T14:35:04.215774+0000","last_became_peered":"2024-01-15T14:35:04.215774+0000","last_unstale":"2024-01-15T14:35:06.220762+0000","last_undegraded":"2024-01-15T14:35:06.220762+0000","last_fullsized":"2024-01-15T14:35:06.220762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038274+0000","last_change":"2024-01-15T14:35:05.243121+0000","last_active":"2024-01-15T14:35:07.038274+0000","last_peered":"2024-01-15T14:35:07.038274+0000","last_clean":"2024-01-15T14:35:07.038274+0000","last_became_active":"2024-01-15T14:35:05.242858+0000","last_became_peered":"2024-01-15T14:35:05.242858+0000","last_unstale":"2024-01-15T14:35:07.038274+0000","last_undegraded":"2024-01-15T14:35:07.038274+0000","last_fullsized":"2024-01-15T14:35:07.038274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:38.527435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222109+0000","last_change":"2024-01-15T14:35:04.214194+0000","last_active":"2024-01-15T14:35:06.222109+0000","last_peered":"2024-01-15T14:35:06.222109+0000","last_clean":"2024-01-15T14:35:06.222109+0000","last_became_active":"2024-01-15T14:35:04.213952+0000","last_became_peered":"2024-01-15T14:35:04.213952+0000","last_unstale":"2024-01-15T14:35:06.222109+0000","last_undegraded":"2024-01-15T14:35:06.222109+0000","last_fullsized":"2024-01-15T14:35:06.222109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914552+0000","last_change":"2024-01-15T14:35:05.237633+0000","last_active":"2024-01-15T14:35:06.914552+0000","last_peered":"2024-01-15T14:35:06.914552+0000","last_clean":"2024-01-15T14:35:06.914552+0000","last_became_active":"2024-01-15T14:35:05.234559+0000","last_became_peered":"2024-01-15T14:35:05.234559+0000","last_unstale":"2024-01-15T14:35:06.914552+0000","last_undegraded":"2024-01-15T14:35:06.914552+0000","last_fullsized":"2024-01-15T14:35:06.914552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220477+0000","last_change":"2024-01-15T14:35:04.212481+0000","last_active":"2024-01-15T14:35:06.220477+0000","last_peered":"2024-01-15T14:35:06.220477+0000","last_clean":"2024-01-15T14:35:06.220477+0000","last_became_active":"2024-01-15T14:35:04.212205+0000","last_became_peered":"2024-01-15T14:35:04.212205+0000","last_unstale":"2024-01-15T14:35:06.220477+0000","last_undegraded":"2024-01-15T14:35:06.220477+0000","last_fullsized":"2024-01-15T14:35:06.220477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220657+0000","last_change":"2024-01-15T14:35:05.229905+0000","last_active":"2024-01-15T14:35:06.220657+0000","last_peered":"2024-01-15T14:35:06.220657+0000","last_clean":"2024-01-15T14:35:06.220657+0000","last_became_active":"2024-01-15T14:35:05.229591+0000","last_became_peered":"2024-01-15T14:35:05.229591+0000","last_unstale":"2024-01-15T14:35:06.220657+0000","last_undegraded":"2024-01-15T14:35:06.220657+0000","last_fullsized":"2024-01-15T14:35:06.220657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337590+0000","last_change":"2024-01-15T14:35:04.212238+0000","last_active":"2024-01-15T14:35:06.337590+0000","last_peered":"2024-01-15T14:35:06.337590+0000","last_clean":"2024-01-15T14:35:06.337590+0000","last_became_active":"2024-01-15T14:35:04.211931+0000","last_became_peered":"2024-01-15T14:35:04.211931+0000","last_unstale":"2024-01-15T14:35:06.337590+0000","last_undegraded":"2024-01-15T14:35:06.337590+0000","last_fullsized":"2024-01-15T14:35:06.337590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:39.371648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038653+0000","last_change":"2024-01-15T14:35:05.215883+0000","last_active":"2024-01-15T14:35:07.038653+0000","last_peered":"2024-01-15T14:35:07.038653+0000","last_clean":"2024-01-15T14:35:07.038653+0000","last_became_active":"2024-01-15T14:35:05.215584+0000","last_became_peered":"2024-01-15T14:35:05.215584+0000","last_unstale":"2024-01-15T14:35:07.038653+0000","last_undegraded":"2024-01-15T14:35:07.038653+0000","last_fullsized":"2024-01-15T14:35:07.038653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222538+0000","last_change":"2024-01-15T14:35:04.212959+0000","last_active":"2024-01-15T14:35:06.222538+0000","last_peered":"2024-01-15T14:35:06.222538+0000","last_clean":"2024-01-15T14:35:06.222538+0000","last_became_active":"2024-01-15T14:35:04.212632+0000","last_became_peered":"2024-01-15T14:35:04.212632+0000","last_unstale":"2024-01-15T14:35:06.222538+0000","last_undegraded":"2024-01-15T14:35:06.222538+0000","last_fullsized":"2024-01-15T14:35:06.222538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337983+0000","last_change":"2024-01-15T14:35:05.232646+0000","last_active":"2024-01-15T14:35:06.337983+0000","last_peered":"2024-01-15T14:35:06.337983+0000","last_clean":"2024-01-15T14:35:06.337983+0000","last_became_active":"2024-01-15T14:35:05.232360+0000","last_became_peered":"2024-01-15T14:35:05.232360+0000","last_unstale":"2024-01-15T14:35:06.337983+0000","last_undegraded":"2024-01-15T14:35:06.337983+0000","last_fullsized":"2024-01-15T14:35:06.337983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337788+0000","last_change":"2024-01-15T14:35:04.207794+0000","last_active":"2024-01-15T14:35:06.337788+0000","last_peered":"2024-01-15T14:35:06.337788+0000","last_clean":"2024-01-15T14:35:06.337788+0000","last_became_active":"2024-01-15T14:35:04.207503+0000","last_became_peered":"2024-01-15T14:35:04.207503+0000","last_unstale":"2024-01-15T14:35:06.337788+0000","last_undegraded":"2024-01-15T14:35:06.337788+0000","last_fullsized":"2024-01-15T14:35:06.337788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914599+0000","last_change":"2024-01-15T14:35:05.228000+0000","last_active":"2024-01-15T14:35:06.914599+0000","last_peered":"2024-01-15T14:35:06.914599+0000","last_clean":"2024-01-15T14:35:06.914599+0000","last_became_active":"2024-01-15T14:35:05.227713+0000","last_became_peered":"2024-01-15T14:35:05.227713+0000","last_unstale":"2024-01-15T14:35:06.914599+0000","last_undegraded":"2024-01-15T14:35:06.914599+0000","last_fullsized":"2024-01-15T14:35:06.914599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038760+0000","last_change":"2024-01-15T14:35:04.212694+0000","last_active":"2024-01-15T14:35:07.038760+0000","last_peered":"2024-01-15T14:35:07.038760+0000","last_clean":"2024-01-15T14:35:07.038760+0000","last_became_active":"2024-01-15T14:35:04.212368+0000","last_became_peered":"2024-01-15T14:35:04.212368+0000","last_unstale":"2024-01-15T14:35:07.038760+0000","last_undegraded":"2024-01-15T14:35:07.038760+0000","last_fullsized":"2024-01-15T14:35:07.038760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038591+0000","last_change":"2024-01-15T14:35:05.235186+0000","last_active":"2024-01-15T14:35:07.038591+0000","last_peered":"2024-01-15T14:35:07.038591+0000","last_clean":"2024-01-15T14:35:07.038591+0000","last_became_active":"2024-01-15T14:35:05.234887+0000","last_became_peered":"2024-01-15T14:35:05.234887+0000","last_unstale":"2024-01-15T14:35:07.038591+0000","last_undegraded":"2024-01-15T14:35:07.038591+0000","last_fullsized":"2024-01-15T14:35:07.038591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:43.060852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222002+0000","last_change":"2024-01-15T14:35:04.212306+0000","last_active":"2024-01-15T14:35:06.222002+0000","last_peered":"2024-01-15T14:35:06.222002+0000","last_clean":"2024-01-15T14:35:06.222002+0000","last_became_active":"2024-01-15T14:35:04.211871+0000","last_became_peered":"2024-01-15T14:35:04.211871+0000","last_unstale":"2024-01-15T14:35:06.222002+0000","last_undegraded":"2024-01-15T14:35:06.222002+0000","last_fullsized":"2024-01-15T14:35:06.222002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223019+0000","last_change":"2024-01-15T14:35:05.242305+0000","last_active":"2024-01-15T14:35:06.223019+0000","last_peered":"2024-01-15T14:35:06.223019+0000","last_clean":"2024-01-15T14:35:06.223019+0000","last_became_active":"2024-01-15T14:35:05.242026+0000","last_became_peered":"2024-01-15T14:35:05.242026+0000","last_unstale":"2024-01-15T14:35:06.223019+0000","last_undegraded":"2024-01-15T14:35:06.223019+0000","last_fullsized":"2024-01-15T14:35:06.223019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914403+0000","last_change":"2024-01-15T14:35:04.205419+0000","last_active":"2024-01-15T14:35:06.914403+0000","last_peered":"2024-01-15T14:35:06.914403+0000","last_clean":"2024-01-15T14:35:06.914403+0000","last_became_active":"2024-01-15T14:35:04.205106+0000","last_became_peered":"2024-01-15T14:35:04.205106+0000","last_unstale":"2024-01-15T14:35:06.914403+0000","last_undegraded":"2024-01-15T14:35:06.914403+0000","last_fullsized":"2024-01-15T14:35:06.914403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914366+0000","last_change":"2024-01-15T14:35:05.239426+0000","last_active":"2024-01-15T14:35:06.914366+0000","last_peered":"2024-01-15T14:35:06.914366+0000","last_clean":"2024-01-15T14:35:06.914366+0000","last_became_active":"2024-01-15T14:35:05.239139+0000","last_became_peered":"2024-01-15T14:35:05.239139+0000","last_unstale":"2024-01-15T14:35:06.914366+0000","last_undegraded":"2024-01-15T14:35:06.914366+0000","last_fullsized":"2024-01-15T14:35:06.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338008+0000","last_change":"2024-01-15T14:35:04.207140+0000","last_active":"2024-01-15T14:35:06.338008+0000","last_peered":"2024-01-15T14:35:06.338008+0000","last_clean":"2024-01-15T14:35:06.338008+0000","last_became_active":"2024-01-15T14:35:04.206804+0000","last_became_peered":"2024-01-15T14:35:04.206804+0000","last_unstale":"2024-01-15T14:35:06.338008+0000","last_undegraded":"2024-01-15T14:35:06.338008+0000","last_fullsized":"2024-01-15T14:35:06.338008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:30.291286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222686+0000","last_change":"2024-01-15T14:35:05.240267+0000","last_active":"2024-01-15T14:35:06.222686+0000","last_peered":"2024-01-15T14:35:06.222686+0000","last_clean":"2024-01-15T14:35:06.222686+0000","last_became_active":"2024-01-15T14:35:05.239993+0000","last_became_peered":"2024-01-15T14:35:05.239993+0000","last_unstale":"2024-01-15T14:35:06.222686+0000","last_undegraded":"2024-01-15T14:35:06.222686+0000","last_fullsized":"2024-01-15T14:35:06.222686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221632+0000","last_change":"2024-01-15T14:35:04.214296+0000","last_active":"2024-01-15T14:35:06.221632+0000","last_peered":"2024-01-15T14:35:06.221632+0000","last_clean":"2024-01-15T14:35:06.221632+0000","last_became_active":"2024-01-15T14:35:04.213992+0000","last_became_peered":"2024-01-15T14:35:04.213992+0000","last_unstale":"2024-01-15T14:35:06.221632+0000","last_undegraded":"2024-01-15T14:35:06.221632+0000","last_fullsized":"2024-01-15T14:35:06.221632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221677+0000","last_change":"2024-01-15T14:35:05.231106+0000","last_active":"2024-01-15T14:35:06.221677+0000","last_peered":"2024-01-15T14:35:06.221677+0000","last_clean":"2024-01-15T14:35:06.221677+0000","last_became_active":"2024-01-15T14:35:05.230594+0000","last_became_peered":"2024-01-15T14:35:05.230594+0000","last_unstale":"2024-01-15T14:35:06.221677+0000","last_undegraded":"2024-01-15T14:35:06.221677+0000","last_fullsized":"2024-01-15T14:35:06.221677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420056+0000","last_change":"2024-01-15T14:35:04.211246+0000","last_active":"2024-01-15T14:35:06.420056+0000","last_peered":"2024-01-15T14:35:06.420056+0000","last_clean":"2024-01-15T14:35:06.420056+0000","last_became_active":"2024-01-15T14:35:04.210971+0000","last_became_peered":"2024-01-15T14:35:04.210971+0000","last_unstale":"2024-01-15T14:35:06.420056+0000","last_undegraded":"2024-01-15T14:35:06.420056+0000","last_fullsized":"2024-01-15T14:35:06.420056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:06.847713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222962+0000","last_change":"2024-01-15T14:35:05.240373+0000","last_active":"2024-01-15T14:35:06.222962+0000","last_peered":"2024-01-15T14:35:06.222962+0000","last_clean":"2024-01-15T14:35:06.222962+0000","last_became_active":"2024-01-15T14:35:05.240078+0000","last_became_peered":"2024-01-15T14:35:05.240078+0000","last_unstale":"2024-01-15T14:35:06.222962+0000","last_undegraded":"2024-01-15T14:35:06.222962+0000","last_fullsized":"2024-01-15T14:35:06.222962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038437+0000","last_change":"2024-01-15T14:35:04.204121+0000","last_active":"2024-01-15T14:35:07.038437+0000","last_peered":"2024-01-15T14:35:07.038437+0000","last_clean":"2024-01-15T14:35:07.038437+0000","last_became_active":"2024-01-15T14:35:04.203811+0000","last_became_peered":"2024-01-15T14:35:04.203811+0000","last_unstale":"2024-01-15T14:35:07.038437+0000","last_undegraded":"2024-01-15T14:35:07.038437+0000","last_fullsized":"2024-01-15T14:35:07.038437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222934+0000","last_change":"2024-01-15T14:35:05.239960+0000","last_active":"2024-01-15T14:35:06.222934+0000","last_peered":"2024-01-15T14:35:06.222934+0000","last_clean":"2024-01-15T14:35:06.222934+0000","last_became_active":"2024-01-15T14:35:05.239645+0000","last_became_peered":"2024-01-15T14:35:05.239645+0000","last_unstale":"2024-01-15T14:35:06.222934+0000","last_undegraded":"2024-01-15T14:35:06.222934+0000","last_fullsized":"2024-01-15T14:35:06.222934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038563+0000","last_change":"2024-01-15T14:35:04.212055+0000","last_active":"2024-01-15T14:35:07.038563+0000","last_peered":"2024-01-15T14:35:07.038563+0000","last_clean":"2024-01-15T14:35:07.038563+0000","last_became_active":"2024-01-15T14:35:04.211788+0000","last_became_peered":"2024-01-15T14:35:04.211788+0000","last_unstale":"2024-01-15T14:35:07.038563+0000","last_undegraded":"2024-01-15T14:35:07.038563+0000","last_fullsized":"2024-01-15T14:35:07.038563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338047+0000","last_change":"2024-01-15T14:35:05.220108+0000","last_active":"2024-01-15T14:35:06.338047+0000","last_peered":"2024-01-15T14:35:06.338047+0000","last_clean":"2024-01-15T14:35:06.338047+0000","last_became_active":"2024-01-15T14:35:05.219812+0000","last_became_peered":"2024-01-15T14:35:05.219812+0000","last_unstale":"2024-01-15T14:35:06.338047+0000","last_undegraded":"2024-01-15T14:35:06.338047+0000","last_fullsized":"2024-01-15T14:35:06.338047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337937+0000","last_change":"2024-01-15T14:35:04.212432+0000","last_active":"2024-01-15T14:35:06.337937+0000","last_peered":"2024-01-15T14:35:06.337937+0000","last_clean":"2024-01-15T14:35:06.337937+0000","last_became_active":"2024-01-15T14:35:04.211351+0000","last_became_peered":"2024-01-15T14:35:04.211351+0000","last_unstale":"2024-01-15T14:35:06.337937+0000","last_undegraded":"2024-01-15T14:35:06.337937+0000","last_fullsized":"2024-01-15T14:35:06.337937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:38.895254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038566+0000","last_change":"2024-01-15T14:35:05.234938+0000","last_active":"2024-01-15T14:35:07.038566+0000","last_peered":"2024-01-15T14:35:07.038566+0000","last_clean":"2024-01-15T14:35:07.038566+0000","last_became_active":"2024-01-15T14:35:05.234597+0000","last_became_peered":"2024-01-15T14:35:05.234597+0000","last_unstale":"2024-01-15T14:35:07.038566+0000","last_undegraded":"2024-01-15T14:35:07.038566+0000","last_fullsized":"2024-01-15T14:35:07.038566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:00.256458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038392+0000","last_change":"2024-01-15T14:35:05.211465+0000","last_active":"2024-01-15T14:35:07.038392+0000","last_peered":"2024-01-15T14:35:07.038392+0000","last_clean":"2024-01-15T14:35:07.038392+0000","last_became_active":"2024-01-15T14:35:05.211188+0000","last_became_peered":"2024-01-15T14:35:05.211188+0000","last_unstale":"2024-01-15T14:35:07.038392+0000","last_undegraded":"2024-01-15T14:35:07.038392+0000","last_fullsized":"2024-01-15T14:35:07.038392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.419968+0000","last_change":"2024-01-15T14:35:04.208525+0000","last_active":"2024-01-15T14:35:06.419968+0000","last_peered":"2024-01-15T14:35:06.419968+0000","last_clean":"2024-01-15T14:35:06.419968+0000","last_became_active":"2024-01-15T14:35:04.208217+0000","last_became_peered":"2024-01-15T14:35:04.208217+0000","last_unstale":"2024-01-15T14:35:06.419968+0000","last_undegraded":"2024-01-15T14:35:06.419968+0000","last_fullsized":"2024-01-15T14:35:06.419968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220595+0000","last_change":"2024-01-15T14:35:05.236274+0000","last_active":"2024-01-15T14:35:06.220595+0000","last_peered":"2024-01-15T14:35:06.220595+0000","last_clean":"2024-01-15T14:35:06.220595+0000","last_became_active":"2024-01-15T14:35:05.236000+0000","last_became_peered":"2024-01-15T14:35:05.236000+0000","last_unstale":"2024-01-15T14:35:06.220595+0000","last_undegraded":"2024-01-15T14:35:06.220595+0000","last_fullsized":"2024-01-15T14:35:06.220595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337639+0000","last_change":"2024-01-15T14:35:04.209747+0000","last_active":"2024-01-15T14:35:06.337639+0000","last_peered":"2024-01-15T14:35:06.337639+0000","last_clean":"2024-01-15T14:35:06.337639+0000","last_became_active":"2024-01-15T14:35:04.209450+0000","last_became_peered":"2024-01-15T14:35:04.209450+0000","last_unstale":"2024-01-15T14:35:06.337639+0000","last_undegraded":"2024-01-15T14:35:06.337639+0000","last_fullsized":"2024-01-15T14:35:06.337639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:37:47.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T14:35:05.216923+0000","last_change":"2024-01-15T14:34:54.738502+0000","last_active":"2024-01-15T14:35:05.216923+0000","last_peered":"2024-01-15T14:35:05.216923+0000","last_clean":"2024-01-15T14:35:05.216923+0000","last_became_active":"2024-01-15T14:34:54.737354+0000","last_became_peered":"2024-01-15T14:34:54.737354+0000","last_unstale":"2024-01-15T14:35:05.216923+0000","last_undegraded":"2024-01-15T14:35:05.216923+0000","last_fullsized":"2024-01-15T14:35:05.216923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_clean_scrub_stamp":"2024-01-15T14:34:53.124419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:09:14.038173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914075+0000","last_change":"2024-01-15T14:35:05.231021+0000","last_active":"2024-01-15T14:35:06.914075+0000","last_peered":"2024-01-15T14:35:06.914075+0000","last_clean":"2024-01-15T14:35:06.914075+0000","last_became_active":"2024-01-15T14:35:05.230721+0000","last_became_peered":"2024-01-15T14:35:05.230721+0000","last_unstale":"2024-01-15T14:35:06.914075+0000","last_undegraded":"2024-01-15T14:35:06.914075+0000","last_fullsized":"2024-01-15T14:35:06.914075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:15.565484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222435+0000","last_change":"2024-01-15T14:35:04.216271+0000","last_active":"2024-01-15T14:35:06.222435+0000","last_peered":"2024-01-15T14:35:06.222435+0000","last_clean":"2024-01-15T14:35:06.222435+0000","last_became_active":"2024-01-15T14:35:04.215955+0000","last_became_peered":"2024-01-15T14:35:04.215955+0000","last_unstale":"2024-01-15T14:35:06.222435+0000","last_undegraded":"2024-01-15T14:35:06.222435+0000","last_fullsized":"2024-01-15T14:35:06.222435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223126+0000","last_change":"2024-01-15T14:35:04.218008+0000","last_active":"2024-01-15T14:35:06.223126+0000","last_peered":"2024-01-15T14:35:06.223126+0000","last_clean":"2024-01-15T14:35:06.223126+0000","last_became_active":"2024-01-15T14:35:04.217672+0000","last_became_peered":"2024-01-15T14:35:04.217672+0000","last_unstale":"2024-01-15T14:35:06.223126+0000","last_undegraded":"2024-01-15T14:35:06.223126+0000","last_fullsized":"2024-01-15T14:35:06.223126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:25.883807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038622+0000","last_change":"2024-01-15T14:35:05.213577+0000","last_active":"2024-01-15T14:35:07.038622+0000","last_peered":"2024-01-15T14:35:07.038622+0000","last_clean":"2024-01-15T14:35:07.038622+0000","last_became_active":"2024-01-15T14:35:05.212565+0000","last_became_peered":"2024-01-15T14:35:05.212565+0000","last_unstale":"2024-01-15T14:35:07.038622+0000","last_undegraded":"2024-01-15T14:35:07.038622+0000","last_fullsized":"2024-01-15T14:35:07.038622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038646+0000","last_change":"2024-01-15T14:35:04.216515+0000","last_active":"2024-01-15T14:35:07.038646+0000","last_peered":"2024-01-15T14:35:07.038646+0000","last_clean":"2024-01-15T14:35:07.038646+0000","last_became_active":"2024-01-15T14:35:04.216291+0000","last_became_peered":"2024-01-15T14:35:04.216291+0000","last_unstale":"2024-01-15T14:35:07.038646+0000","last_undegraded":"2024-01-15T14:35:07.038646+0000","last_fullsized":"2024-01-15T14:35:07.038646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221002+0000","last_change":"2024-01-15T14:35:05.233144+0000","last_active":"2024-01-15T14:35:06.221002+0000","last_peered":"2024-01-15T14:35:06.221002+0000","last_clean":"2024-01-15T14:35:06.221002+0000","last_became_active":"2024-01-15T14:35:05.232813+0000","last_became_peered":"2024-01-15T14:35:05.232813+0000","last_unstale":"2024-01-15T14:35:06.221002+0000","last_undegraded":"2024-01-15T14:35:06.221002+0000","last_fullsized":"2024-01-15T14:35:06.221002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222447+0000","last_change":"2024-01-15T14:35:04.215265+0000","last_active":"2024-01-15T14:35:06.222447+0000","last_peered":"2024-01-15T14:35:06.222447+0000","last_clean":"2024-01-15T14:35:06.222447+0000","last_became_active":"2024-01-15T14:35:04.214951+0000","last_became_peered":"2024-01-15T14:35:04.214951+0000","last_unstale":"2024-01-15T14:35:06.222447+0000","last_undegraded":"2024-01-15T14:35:06.222447+0000","last_fullsized":"2024-01-15T14:35:06.222447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914377+0000","last_change":"2024-01-15T14:35:05.228381+0000","last_active":"2024-01-15T14:35:06.914377+0000","last_peered":"2024-01-15T14:35:06.914377+0000","last_clean":"2024-01-15T14:35:06.914377+0000","last_became_active":"2024-01-15T14:35:05.228101+0000","last_became_peered":"2024-01-15T14:35:05.228101+0000","last_unstale":"2024-01-15T14:35:06.914377+0000","last_undegraded":"2024-01-15T14:35:06.914377+0000","last_fullsized":"2024-01-15T14:35:06.914377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222345+0000","last_change":"2024-01-15T14:35:04.215917+0000","last_active":"2024-01-15T14:35:06.222345+0000","last_peered":"2024-01-15T14:35:06.222345+0000","last_clean":"2024-01-15T14:35:06.222345+0000","last_became_active":"2024-01-15T14:35:04.215592+0000","last_became_peered":"2024-01-15T14:35:04.215592+0000","last_unstale":"2024-01-15T14:35:06.222345+0000","last_undegraded":"2024-01-15T14:35:06.222345+0000","last_fullsized":"2024-01-15T14:35:06.222345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038319+0000","last_change":"2024-01-15T14:35:05.233251+0000","last_active":"2024-01-15T14:35:07.038319+0000","last_peered":"2024-01-15T14:35:07.038319+0000","last_clean":"2024-01-15T14:35:07.038319+0000","last_became_active":"2024-01-15T14:35:05.232982+0000","last_became_peered":"2024-01-15T14:35:05.232982+0000","last_unstale":"2024-01-15T14:35:07.038319+0000","last_undegraded":"2024-01-15T14:35:07.038319+0000","last_fullsized":"2024-01-15T14:35:07.038319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:15.827108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038435+0000","last_change":"2024-01-15T14:35:04.218539+0000","last_active":"2024-01-15T14:35:07.038435+0000","last_peered":"2024-01-15T14:35:07.038435+0000","last_clean":"2024-01-15T14:35:07.038435+0000","last_became_active":"2024-01-15T14:35:04.218307+0000","last_became_peered":"2024-01-15T14:35:04.218307+0000","last_unstale":"2024-01-15T14:35:07.038435+0000","last_undegraded":"2024-01-15T14:35:07.038435+0000","last_fullsized":"2024-01-15T14:35:07.038435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220625+0000","last_change":"2024-01-15T14:35:05.220403+0000","last_active":"2024-01-15T14:35:06.220625+0000","last_peered":"2024-01-15T14:35:06.220625+0000","last_clean":"2024-01-15T14:35:06.220625+0000","last_became_active":"2024-01-15T14:35:05.220033+0000","last_became_peered":"2024-01-15T14:35:05.220033+0000","last_unstale":"2024-01-15T14:35:06.220625+0000","last_undegraded":"2024-01-15T14:35:06.220625+0000","last_fullsized":"2024-01-15T14:35:06.220625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038582+0000","last_change":"2024-01-15T14:35:04.211061+0000","last_active":"2024-01-15T14:35:07.038582+0000","last_peered":"2024-01-15T14:35:07.038582+0000","last_clean":"2024-01-15T14:35:07.038582+0000","last_became_active":"2024-01-15T14:35:04.210748+0000","last_became_peered":"2024-01-15T14:35:04.210748+0000","last_unstale":"2024-01-15T14:35:07.038582+0000","last_undegraded":"2024-01-15T14:35:07.038582+0000","last_fullsized":"2024-01-15T14:35:07.038582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220819+0000","last_change":"2024-01-15T14:35:05.232247+0000","last_active":"2024-01-15T14:35:06.220819+0000","last_peered":"2024-01-15T14:35:06.220819+0000","last_clean":"2024-01-15T14:35:06.220819+0000","last_became_active":"2024-01-15T14:35:05.231914+0000","last_became_peered":"2024-01-15T14:35:05.231914+0000","last_unstale":"2024-01-15T14:35:06.220819+0000","last_undegraded":"2024-01-15T14:35:06.220819+0000","last_fullsized":"2024-01-15T14:35:06.220819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222720+0000","last_change":"2024-01-15T14:35:04.216586+0000","last_active":"2024-01-15T14:35:06.222720+0000","last_peered":"2024-01-15T14:35:06.222720+0000","last_clean":"2024-01-15T14:35:06.222720+0000","last_became_active":"2024-01-15T14:35:04.216231+0000","last_became_peered":"2024-01-15T14:35:04.216231+0000","last_unstale":"2024-01-15T14:35:06.222720+0000","last_undegraded":"2024-01-15T14:35:06.222720+0000","last_fullsized":"2024-01-15T14:35:06.222720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221812+0000","last_change":"2024-01-15T14:35:05.237033+0000","last_active":"2024-01-15T14:35:06.221812+0000","last_peered":"2024-01-15T14:35:06.221812+0000","last_clean":"2024-01-15T14:35:06.221812+0000","last_became_active":"2024-01-15T14:35:05.236738+0000","last_became_peered":"2024-01-15T14:35:05.236738+0000","last_unstale":"2024-01-15T14:35:06.221812+0000","last_undegraded":"2024-01-15T14:35:06.221812+0000","last_fullsized":"2024-01-15T14:35:06.221812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420017+0000","last_change":"2024-01-15T14:35:04.208500+0000","last_active":"2024-01-15T14:35:06.420017+0000","last_peered":"2024-01-15T14:35:06.420017+0000","last_clean":"2024-01-15T14:35:06.420017+0000","last_became_active":"2024-01-15T14:35:04.208227+0000","last_became_peered":"2024-01-15T14:35:04.208227+0000","last_unstale":"2024-01-15T14:35:06.420017+0000","last_undegraded":"2024-01-15T14:35:06.420017+0000","last_fullsized":"2024-01-15T14:35:06.420017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:11:11.486899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223119+0000","last_change":"2024-01-15T14:35:05.232373+0000","last_active":"2024-01-15T14:35:06.223119+0000","last_peered":"2024-01-15T14:35:06.223119+0000","last_clean":"2024-01-15T14:35:06.223119+0000","last_became_active":"2024-01-15T14:35:05.232098+0000","last_became_peered":"2024-01-15T14:35:05.232098+0000","last_unstale":"2024-01-15T14:35:06.223119+0000","last_undegraded":"2024-01-15T14:35:06.223119+0000","last_fullsized":"2024-01-15T14:35:06.223119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038754+0000","last_change":"2024-01-15T14:35:04.213296+0000","last_active":"2024-01-15T14:35:07.038754+0000","last_peered":"2024-01-15T14:35:07.038754+0000","last_clean":"2024-01-15T14:35:07.038754+0000","last_became_active":"2024-01-15T14:35:04.212955+0000","last_became_peered":"2024-01-15T14:35:04.212955+0000","last_unstale":"2024-01-15T14:35:07.038754+0000","last_undegraded":"2024-01-15T14:35:07.038754+0000","last_fullsized":"2024-01-15T14:35:07.038754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:10.265464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338036+0000","last_change":"2024-01-15T14:35:05.219979+0000","last_active":"2024-01-15T14:35:06.338036+0000","last_peered":"2024-01-15T14:35:06.338036+0000","last_clean":"2024-01-15T14:35:06.338036+0000","last_became_active":"2024-01-15T14:35:05.219668+0000","last_became_peered":"2024-01-15T14:35:05.219668+0000","last_unstale":"2024-01-15T14:35:06.338036+0000","last_undegraded":"2024-01-15T14:35:06.338036+0000","last_fullsized":"2024-01-15T14:35:06.338036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420118+0000","last_change":"2024-01-15T14:35:04.208970+0000","last_active":"2024-01-15T14:35:06.420118+0000","last_peered":"2024-01-15T14:35:06.420118+0000","last_clean":"2024-01-15T14:35:06.420118+0000","last_became_active":"2024-01-15T14:35:04.208699+0000","last_became_peered":"2024-01-15T14:35:04.208699+0000","last_unstale":"2024-01-15T14:35:06.420118+0000","last_undegraded":"2024-01-15T14:35:06.420118+0000","last_fullsized":"2024-01-15T14:35:06.420118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223420+0000","last_change":"2024-01-15T14:35:05.239873+0000","last_active":"2024-01-15T14:35:06.223420+0000","last_peered":"2024-01-15T14:35:06.223420+0000","last_clean":"2024-01-15T14:35:06.223420+0000","last_became_active":"2024-01-15T14:35:05.239571+0000","last_became_peered":"2024-01-15T14:35:05.239571+0000","last_unstale":"2024-01-15T14:35:06.223420+0000","last_undegraded":"2024-01-15T14:35:06.223420+0000","last_fullsized":"2024-01-15T14:35:06.223420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221810+0000","last_change":"2024-01-15T14:35:05.230887+0000","last_active":"2024-01-15T14:35:06.221810+0000","last_peered":"2024-01-15T14:35:06.221810+0000","last_clean":"2024-01-15T14:35:06.221810+0000","last_became_active":"2024-01-15T14:35:05.230187+0000","last_became_peered":"2024-01-15T14:35:05.230187+0000","last_unstale":"2024-01-15T14:35:06.221810+0000","last_undegraded":"2024-01-15T14:35:06.221810+0000","last_fullsized":"2024-01-15T14:35:06.221810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420110+0000","last_change":"2024-01-15T14:35:05.217285+0000","last_active":"2024-01-15T14:35:06.420110+0000","last_peered":"2024-01-15T14:35:06.420110+0000","last_clean":"2024-01-15T14:35:06.420110+0000","last_became_active":"2024-01-15T14:35:05.217015+0000","last_became_peered":"2024-01-15T14:35:05.217015+0000","last_unstale":"2024-01-15T14:35:06.420110+0000","last_undegraded":"2024-01-15T14:35:06.420110+0000","last_fullsized":"2024-01-15T14:35:06.420110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337918+0000","last_change":"2024-01-15T14:35:05.220986+0000","last_active":"2024-01-15T14:35:06.337918+0000","last_peered":"2024-01-15T14:35:06.337918+0000","last_clean":"2024-01-15T14:35:06.337918+0000","last_became_active":"2024-01-15T14:35:05.220704+0000","last_became_peered":"2024-01-15T14:35:05.220704+0000","last_unstale":"2024-01-15T14:35:06.337918+0000","last_undegraded":"2024-01-15T14:35:06.337918+0000","last_fullsized":"2024-01-15T14:35:06.337918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914367+0000","last_change":"2024-01-15T14:35:05.232048+0000","last_active":"2024-01-15T14:35:06.914367+0000","last_peered":"2024-01-15T14:35:06.914367+0000","last_clean":"2024-01-15T14:35:06.914367+0000","last_became_active":"2024-01-15T14:35:05.231708+0000","last_became_peered":"2024-01-15T14:35:05.231708+0000","last_unstale":"2024-01-15T14:35:06.914367+0000","last_undegraded":"2024-01-15T14:35:06.914367+0000","last_fullsized":"2024-01-15T14:35:06.914367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220904+0000","last_change":"2024-01-15T14:35:05.236304+0000","last_active":"2024-01-15T14:35:06.220904+0000","last_peered":"2024-01-15T14:35:06.220904+0000","last_clean":"2024-01-15T14:35:06.220904+0000","last_became_active":"2024-01-15T14:35:05.235435+0000","last_became_peered":"2024-01-15T14:35:05.235435+0000","last_unstale":"2024-01-15T14:35:06.220904+0000","last_undegraded":"2024-01-15T14:35:06.220904+0000","last_fullsized":"2024-01-15T14:35:06.220904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038669+0000","last_change":"2024-01-15T14:35:05.216092+0000","last_active":"2024-01-15T14:35:07.038669+0000","last_peered":"2024-01-15T14:35:07.038669+0000","last_clean":"2024-01-15T14:35:07.038669+0000","last_became_active":"2024-01-15T14:35:05.215618+0000","last_became_peered":"2024-01-15T14:35:05.215618+0000","last_unstale":"2024-01-15T14:35:07.038669+0000","last_undegraded":"2024-01-15T14:35:07.038669+0000","last_fullsized":"2024-01-15T14:35:07.038669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222044+0000","last_change":"2024-01-15T14:35:05.229045+0000","last_active":"2024-01-15T14:35:06.222044+0000","last_peered":"2024-01-15T14:35:06.222044+0000","last_clean":"2024-01-15T14:35:06.222044+0000","last_became_active":"2024-01-15T14:35:05.228771+0000","last_became_peered":"2024-01-15T14:35:05.228771+0000","last_unstale":"2024-01-15T14:35:06.222044+0000","last_undegraded":"2024-01-15T14:35:06.222044+0000","last_fullsized":"2024-01-15T14:35:06.222044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220972+0000","last_change":"2024-01-15T14:35:05.235711+0000","last_active":"2024-01-15T14:35:06.220972+0000","last_peered":"2024-01-15T14:35:06.220972+0000","last_clean":"2024-01-15T14:35:06.220972+0000","last_became_active":"2024-01-15T14:35:05.235419+0000","last_became_peered":"2024-01-15T14:35:05.235419+0000","last_unstale":"2024-01-15T14:35:06.220972+0000","last_undegraded":"2024-01-15T14:35:06.220972+0000","last_fullsized":"2024-01-15T14:35:06.220972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337825+0000","last_change":"2024-01-15T14:35:05.221416+0000","last_active":"2024-01-15T14:35:06.337825+0000","last_peered":"2024-01-15T14:35:06.337825+0000","last_clean":"2024-01-15T14:35:06.337825+0000","last_became_active":"2024-01-15T14:35:05.221124+0000","last_became_peered":"2024-01-15T14:35:05.221124+0000","last_unstale":"2024-01-15T14:35:06.337825+0000","last_undegraded":"2024-01-15T14:35:06.337825+0000","last_fullsized":"2024-01-15T14:35:06.337825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220561+0000","last_change":"2024-01-15T14:35:05.235785+0000","last_active":"2024-01-15T14:35:06.220561+0000","last_peered":"2024-01-15T14:35:06.220561+0000","last_clean":"2024-01-15T14:35:06.220561+0000","last_became_active":"2024-01-15T14:35:05.235193+0000","last_became_peered":"2024-01-15T14:35:05.235193+0000","last_unstale":"2024-01-15T14:35:06.220561+0000","last_undegraded":"2024-01-15T14:35:06.220561+0000","last_fullsized":"2024-01-15T14:35:06.220561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337729+0000","last_change":"2024-01-15T14:35:05.225944+0000","last_active":"2024-01-15T14:35:06.337729+0000","last_peered":"2024-01-15T14:35:06.337729+0000","last_clean":"2024-01-15T14:35:06.337729+0000","last_became_active":"2024-01-15T14:35:05.225660+0000","last_became_peered":"2024-01-15T14:35:05.225660+0000","last_unstale":"2024-01-15T14:35:06.337729+0000","last_undegraded":"2024-01-15T14:35:06.337729+0000","last_fullsized":"2024-01-15T14:35:06.337729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914286+0000","last_change":"2024-01-15T14:35:05.243644+0000","last_active":"2024-01-15T14:35:06.914286+0000","last_peered":"2024-01-15T14:35:06.914286+0000","last_clean":"2024-01-15T14:35:06.914286+0000","last_became_active":"2024-01-15T14:35:05.243362+0000","last_became_peered":"2024-01-15T14:35:05.243362+0000","last_unstale":"2024-01-15T14:35:06.914286+0000","last_undegraded":"2024-01-15T14:35:06.914286+0000","last_fullsized":"2024-01-15T14:35:06.914286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:58.010415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222177+0000","last_change":"2024-01-15T14:35:05.237833+0000","last_active":"2024-01-15T14:35:06.222177+0000","last_peered":"2024-01-15T14:35:06.222177+0000","last_clean":"2024-01-15T14:35:06.222177+0000","last_became_active":"2024-01-15T14:35:05.237576+0000","last_became_peered":"2024-01-15T14:35:05.237576+0000","last_unstale":"2024-01-15T14:35:06.222177+0000","last_undegraded":"2024-01-15T14:35:06.222177+0000","last_fullsized":"2024-01-15T14:35:06.222177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T14:35:09.696 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T14:35:09.698 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-15T14:35:09.698 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-15T14:35:09.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-15T14:35:09.698 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-15T14:35:09.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 Processor -- start 2024-01-15T14:35:09.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 -- start start 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c15e080 0x7f566c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f566c0ce4d0 0x7f566c0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c0d0150 con 0x7f566c0ce910 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c15e460 con 0x7f566c0ce4d0 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5672e85700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0cee50 con 0x7f566c15e080 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f566bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f566c0ce4d0 0x7f566c0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.854+0000 7f5671422700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f566c0ce910 0x7f566c0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38586/0 (socket says 172.21.15.77:38586) 2024-01-15T14:35:09.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f5671422700 1 -- 172.21.15.77:0/1268177569 learned_addr learned my addr 172.21.15.77:0/1268177569 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f5670c21700 1 --2- 172.21.15.77:0/1268177569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c15e080 0x7f566c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566bfff700 1 -- 172.21.15.77:0/1268177569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c15e080 msgr2=0x7f566c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566bfff700 1 --2- 172.21.15.77:0/1268177569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c15e080 0x7f566c0cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566bfff700 1 -- 172.21.15.77:0/1268177569 >> v1:172.21.15.77:6789/0 conn(0x7f566c0ce910 legacy=0x7f566c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566bfff700 1 -- 172.21.15.77:0/1268177569 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c0cf130 con 0x7f566c0ce4d0 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566bfff700 1 --2- 172.21.15.77:0/1268177569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c0ce4d0 0x7f566c0d1be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5658009840 tx=0x7f5658009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9012e52550baccab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566b7fe700 1 -- 172.21.15.77:0/1268177569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658011830 con 0x7f566c0ce4d0 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566b7fe700 1 -- 172.21.15.77:0/1268177569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5658011990 con 0x7f566c0ce4d0 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.855+0000 7f566b7fe700 1 -- 172.21.15.77:0/1268177569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658011d20 con 0x7f566c0ce4d0 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 -- 172.21.15.77:0/1268177569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c0ce4d0 msgr2=0x7f566c0d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 --2- 172.21.15.77:0/1268177569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c0ce4d0 0x7f566c0d1be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5658009840 tx=0x7f5658009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:09.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 -- 172.21.15.77:0/1268177569 shutdown_connections 2024-01-15T14:35:09.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 --2- 172.21.15.77:0/1268177569 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c15e080 0x7f566c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 --2- 172.21.15.77:0/1268177569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c0ce4d0 0x7f566c0d1be0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 -- 172.21.15.77:0/1268177569 >> 172.21.15.77:0/1268177569 conn(0x7f566c073130 msgr2=0x7f566c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:09.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 -- 172.21.15.77:0/1268177569 shutdown_connections 2024-01-15T14:35:09.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.856+0000 7f5672e85700 1 -- 172.21.15.77:0/1268177569 wait complete. 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.857+0000 7f5672e85700 1 Processor -- start 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.857+0000 7f5672e85700 1 -- start start 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c0ce4d0 0x7f566c1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566c0ce910 0x7f566c1f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 0x7f566c1f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c14a8b0 con 0x7f566c15e080 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c157f30 con 0x7f566c0ce910 2024-01-15T14:35:09.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c14a5d0 con 0x7f566c0ce4d0 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 0x7f566c1f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 0x7f566c1f7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57320/0 (socket says 172.21.15.77:57320) 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 -- 172.21.15.77:0/1722045229 learned_addr learned my addr 172.21.15.77:0/1722045229 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f566bfff700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566c0ce910 0x7f566c1f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5670c21700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c0ce4d0 0x7f566c1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 -- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c0ce4d0 msgr2=0x7f566c1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c0ce4d0 0x7f566c1579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 -- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566c0ce910 msgr2=0x7f566c1f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566c0ce910 0x7f566c1f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 -- 172.21.15.77:0/1722045229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56580094a0 con 0x7f566c15e080 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f566bfff700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566c0ce910 0x7f566c1f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5670c21700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c0ce4d0 0x7f566c1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:09.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5671422700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 0x7f566c1f7840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f566400cb20 tx=0x7f566400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400aac0 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566400ac20 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c0b5270 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664012400 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.858+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c0b5830 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.859+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c0b5a20 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.860+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f566401b3b0 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.860+0000 7f56697fa700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f565006b350 0x7f565006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.860+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5664089710 con 0x7f566c15e080 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.860+0000 7f5670c21700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f565006b350 0x7f565006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:09.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.861+0000 7f5670c21700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f565006b350 0x7f565006d810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5660006fd0 tx=0x7f5660008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:09.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:09.865+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56640546c0 con 0x7f566c15e080 2024-01-15T14:35:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.114+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f566c067f60 con 0x7f566c15e080 2024-01-15T14:35:10.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.117+0000 7f56697fa700 1 -- 172.21.15.77:0/1722045229 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171558 (secure 0 0 0) 0x7f5664057ce0 con 0x7f566c15e080 2024-01-15T14:35:10.120 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:10.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f565006b350 msgr2=0x7f565006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f565006b350 0x7f565006d810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5660006fd0 tx=0x7f5660008040 comp rx=0 tx=0).stop 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 msgr2=0x7f566c1f7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 0x7f566c1f7840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f566400cb20 tx=0x7f566400cee0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 shutdown_connections 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f565006b350 0x7f565006d810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f566c0ce4d0 0x7f566c1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15e080 0x7f566c1f7840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 --2- 172.21.15.77:0/1722045229 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f566c0ce910 0x7f566c1f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 >> 172.21.15.77:0/1722045229 conn(0x7f566c073130 msgr2=0x7f566c0cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 shutdown_connections 2024-01-15T14:35:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.120+0000 7f5672e85700 1 -- 172.21.15.77:0/1722045229 wait complete. 2024-01-15T14:35:10.132 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4256185546},{"type":"v1","addr":"172.21.15.77:6833","nonce":4256185546}]},"active_addr":"172.21.15.77:6833/4256185546","active_change":"2024-01-15T14:34:45.009997+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2308061142}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3243112581}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3952925538}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1605448820}]}]} 2024-01-15T14:35:10.135 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-15T14:35:10.135 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-15T14:35:10.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:10.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.257+0000 7f80d8bfd700 1 Processor -- start 2024-01-15T14:35:10.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.257+0000 7f80d8bfd700 1 -- start start 2024-01-15T14:35:10.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d8bfd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f80d4073f90 0x7f80d40ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d8bfd700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f80d406ee50 0x7f80d4074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d8bfd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4074e90 con 0x7f80d415c140 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d8bfd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d40ce5d0 con 0x7f80d4073f90 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d8bfd700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d40748b0 con 0x7f80d406ee50 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f80d4073f90 0x7f80d40ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d2d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f80d415c140 0x7f80d415f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38592/0 (socket says 172.21.15.77:38592) 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d2d9d700 1 -- 172.21.15.77:0/885185022 learned_addr learned my addr 172.21.15.77:0/885185022 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d1d9b700 1 --2- 172.21.15.77:0/885185022 >> v2:172.21.15.159:3300/0 conn(0x7f80d406ee50 0x7f80d4074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d259c700 1 -- 172.21.15.77:0/885185022 >> v2:172.21.15.159:3300/0 conn(0x7f80d406ee50 msgr2=0x7f80d4074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d259c700 1 --2- 172.21.15.77:0/885185022 >> v2:172.21.15.159:3300/0 conn(0x7f80d406ee50 0x7f80d4074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d259c700 1 -- 172.21.15.77:0/885185022 >> v1:172.21.15.77:6789/0 conn(0x7f80d415c140 legacy=0x7f80d415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d259c700 1 -- 172.21.15.77:0/885185022 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d415c700 con 0x7f80d4073f90 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d159a700 1 -- 172.21.15.77:0/885185022 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79254672 0 0) 0x7f80d4074e90 con 0x7f80d415c140 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d259c700 1 --2- 172.21.15.77:0/885185022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40ce090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009840 tx=0x7f80c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dca38d2b789dd35 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d159a700 1 -- 172.21.15.77:0/885185022 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011830 con 0x7f80d4073f90 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d159a700 1 -- 172.21.15.77:0/885185022 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c4011990 con 0x7f80d4073f90 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.258+0000 7f80d159a700 1 -- 172.21.15.77:0/885185022 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011d20 con 0x7f80d4073f90 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 -- 172.21.15.77:0/885185022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 msgr2=0x7f80d40ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/885185022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40ce090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009840 tx=0x7f80c4009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 -- 172.21.15.77:0/885185022 shutdown_connections 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/885185022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40ce090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/885185022 >> v2:172.21.15.159:3300/0 conn(0x7f80d406ee50 0x7f80d4074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 -- 172.21.15.77:0/885185022 >> 172.21.15.77:0/885185022 conn(0x7f80d40cd570 msgr2=0x7f80d40cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 -- 172.21.15.77:0/885185022 shutdown_connections 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.259+0000 7f80d8bfd700 1 -- 172.21.15.77:0/885185022 wait complete. 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.260+0000 7f80d8bfd700 1 Processor -- start 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.260+0000 7f80d8bfd700 1 -- start start 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d8bfd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80d406ee50 0x7f80d4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d8bfd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d8bfd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80d415c140 0x7f80d40ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d8bfd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d40cb710 con 0x7f80d4073f90 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d8bfd700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d40cbb30 con 0x7f80d406ee50 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d8bfd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d40b5c30 con 0x7f80d415c140 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57346/0 (socket says 172.21.15.77:57346) 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 -- 172.21.15.77:0/508530566 learned_addr learned my addr 172.21.15.77:0/508530566 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d259c700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80d406ee50 0x7f80d4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 -- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80d415c140 msgr2=0x7f80d40ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d2d9d700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80d415c140 0x7f80d40ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80d415c140 0x7f80d40ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 -- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80d406ee50 msgr2=0x7f80d4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80d406ee50 0x7f80d4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d1d9b700 1 -- 172.21.15.77:0/508530566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40094a0 con 0x7f80d4073f90 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d259c700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80d406ee50 0x7f80d4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.261+0000 7f80d2d9d700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80d415c140 0x7f80d40ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.262+0000 7f80d1d9b700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40c9e60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80c800e840 tx=0x7f80c800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.262+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c8013610 con 0x7f80d4073f90 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.262+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c8013770 con 0x7f80d4073f90 2024-01-15T14:35:10.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.262+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d40caa90 con 0x7f80d4073f90 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.262+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c8013b20 con 0x7f80d4073f90 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.262+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d40cb330 con 0x7f80d4073f90 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.263+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d40cae90 con 0x7f80d4073f90 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.263+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f80c8013c80 con 0x7f80d4073f90 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.264+0000 7f80c2ffd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f80b406b350 0x7f80b406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.264+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80c8089810 con 0x7f80d4073f90 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.264+0000 7f80d259c700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f80b406b350 0x7f80b406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.264+0000 7f80d259c700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f80b406b350 0x7f80b406d810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009b90 tx=0x7f80c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:10.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.268+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80c8054670 con 0x7f80d4073f90 2024-01-15T14:35:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.460+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d4067f60 con 0x7f80d4073f90 2024-01-15T14:35:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.461+0000 7f80c2ffd700 1 -- 172.21.15.77:0/508530566 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f80c8057c90 con 0x7f80d4073f90 2024-01-15T14:35:10.461 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:10.462 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f80b406b350 msgr2=0x7f80b406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f80b406b350 0x7f80b406d810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009b90 tx=0x7f80c4008030 comp rx=0 tx=0).stop 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 msgr2=0x7f80d40c9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40c9e60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80c800e840 tx=0x7f80c800ec00 comp rx=0 tx=0).stop 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 shutdown_connections 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f80b406b350 0x7f80b406d810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f80d415c140 0x7f80d40ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f80d406ee50 0x7f80d4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 --2- 172.21.15.77:0/508530566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4073f90 0x7f80d40c9e60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 >> 172.21.15.77:0/508530566 conn(0x7f80d40cd570 msgr2=0x7f80d406e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 shutdown_connections 2024-01-15T14:35:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.463+0000 7f80d8bfd700 1 -- 172.21.15.77:0/508530566 wait complete. 2024-01-15T14:35:10.475 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-15T14:35:10.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.596+0000 7f9413cfb700 1 Processor -- start 2024-01-15T14:35:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9413cfb700 1 -- start start 2024-01-15T14:35:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9413cfb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f940c15e2a0 0x7f940c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9413cfb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c073f90 0x7f940c074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9413cfb700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940c06e7f0 con 0x7f940c0ce090 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9413cfb700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c074960 con 0x7f940c15e2a0 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9413cfb700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c06dec0 con 0x7f940c073f90 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.597+0000 7f9411a97700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f940c15e2a0 0x7f940c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f940c15e2a0 0x7f940c06ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57362/0 (socket says 172.21.15.77:57362) 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 -- 172.21.15.77:0/2695863840 learned_addr learned my addr 172.21.15.77:0/2695863840 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9412298700 1 --2- 172.21.15.77:0/2695863840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c073f90 0x7f940c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 -- 172.21.15.77:0/2695863840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c073f90 msgr2=0x7f940c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 --2- 172.21.15.77:0/2695863840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c073f90 0x7f940c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 -- 172.21.15.77:0/2695863840 >> v1:172.21.15.159:6789/0 conn(0x7f940c0ce090 legacy=0x7f940c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 -- 172.21.15.77:0/2695863840 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c06e1a0 con 0x7f940c15e2a0 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9411a97700 1 --2- 172.21.15.77:0/2695863840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c15e2a0 0x7f940c06ba70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94000098f0 tx=0x7f9400009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc58e4ea6f0b3cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9410a95700 1 -- 172.21.15.77:0/2695863840 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400004230 con 0x7f940c15e2a0 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9410a95700 1 -- 172.21.15.77:0/2695863840 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9400004390 con 0x7f940c15e2a0 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.598+0000 7f9410a95700 1 -- 172.21.15.77:0/2695863840 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000036a0 con 0x7f940c15e2a0 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 -- 172.21.15.77:0/2695863840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c15e2a0 msgr2=0x7f940c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 --2- 172.21.15.77:0/2695863840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c15e2a0 0x7f940c06ba70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94000098f0 tx=0x7f9400009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 -- 172.21.15.77:0/2695863840 shutdown_connections 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 --2- 172.21.15.77:0/2695863840 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c073f90 0x7f940c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 --2- 172.21.15.77:0/2695863840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c15e2a0 0x7f940c06ba70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 -- 172.21.15.77:0/2695863840 >> 172.21.15.77:0/2695863840 conn(0x7f940c0cd570 msgr2=0x7f940c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 -- 172.21.15.77:0/2695863840 shutdown_connections 2024-01-15T14:35:10.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.599+0000 7f9413cfb700 1 -- 172.21.15.77:0/2695863840 wait complete. 2024-01-15T14:35:10.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.600+0000 7f9413cfb700 1 Processor -- start 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.600+0000 7f9413cfb700 1 -- start start 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 0x7f940c159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c0ce090 0x7f940c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f940c15e2a0 0x7f940c15a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c1e6a70 con 0x7f940c073f90 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c159f90 con 0x7f940c15e2a0 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c160a60 con 0x7f940c0ce090 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 0x7f940c159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 0x7f940c159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57376/0 (socket says 172.21.15.77:57376) 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 -- 172.21.15.77:0/1801019374 learned_addr learned my addr 172.21.15.77:0/1801019374 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9412298700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f940c15e2a0 0x7f940c15a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411296700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c0ce090 0x7f940c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 -- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c0ce090 msgr2=0x7f940c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c0ce090 0x7f940c15a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 -- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f940c15e2a0 msgr2=0x7f940c15a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f940c15e2a0 0x7f940c15a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 -- 172.21.15.77:0/1801019374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9400009550 con 0x7f940c073f90 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9411a97700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 0x7f940c159a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9400009130 tx=0x7f9400004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400003e90 con 0x7f940c073f90 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940000b660 con 0x7f940c073f90 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940c1e6d10 con 0x7f940c073f90 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000b9f0 con 0x7f940c073f90 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.601+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940c1e72d0 con 0x7f940c073f90 2024-01-15T14:35:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.602+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940c067f60 con 0x7f940c073f90 2024-01-15T14:35:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.603+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9400021070 con 0x7f940c073f90 2024-01-15T14:35:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.603+0000 7f93fe7fc700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f93f006b350 0x7f93f006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.603+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f940008b470 con 0x7f940c073f90 2024-01-15T14:35:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.603+0000 7f9411296700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f93f006b350 0x7f93f006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.604+0000 7f9411296700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f93f006b350 0x7f93f006d810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9408009870 tx=0x7f9408008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.608+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9400056370 con 0x7f940c073f90 2024-01-15T14:35:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.800+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f940c072e00 con 0x7f940c073f90 2024-01-15T14:35:10.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.801+0000 7f93fe7fc700 1 -- 172.21.15.77:0/1801019374 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f940002ce80 con 0x7f940c073f90 2024-01-15T14:35:10.801 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:10.801 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f93f006b350 msgr2=0x7f93f006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f93f006b350 0x7f93f006d810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9408009870 tx=0x7f9408008040 comp rx=0 tx=0).stop 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 msgr2=0x7f940c159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 0x7f940c159a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9400009130 tx=0x7f9400004750 comp rx=0 tx=0).stop 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 shutdown_connections 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f93f006b350 0x7f93f006d810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f940c15e2a0 0x7f940c15a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f940c073f90 0x7f940c159a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 --2- 172.21.15.77:0/1801019374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f940c0ce090 0x7f940c15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 >> 172.21.15.77:0/1801019374 conn(0x7f940c0cd570 msgr2=0x7f940c06b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 shutdown_connections 2024-01-15T14:35:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:10.803+0000 7f9413cfb700 1 -- 172.21.15.77:0/1801019374 wait complete. 2024-01-15T14:35:10.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-15T14:35:10.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-15T14:35:10.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-15T14:35:10.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-15T14:35:10.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-15T14:35:10.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-15T14:35:10.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-15T14:35:10.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-15T14:35:11.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.023+0000 7f7787428700 1 Processor -- start 2024-01-15T14:35:11.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.023+0000 7f7787428700 1 -- start start 2024-01-15T14:35:11.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.024+0000 7fb328c6a700 1 Processor -- start 2024-01-15T14:35:11.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f7787428700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f77800cf440 0x7f778006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f7787428700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 0x7f77800b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f7787428700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7780072110 con 0x7f7780070fd0 2024-01-15T14:35:11.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f7787428700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778006ee00 con 0x7f77800cf440 2024-01-15T14:35:11.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f7787428700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7780071500 con 0x7f77801665a0 2024-01-15T14:35:11.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7fb328c6a700 1 -- start start 2024-01-15T14:35:11.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f77859c5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 0x7f77800b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f77859c5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 0x7f77800b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53660/0 (socket says 172.21.15.77:53660) 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f77859c5700 1 -- 172.21.15.77:0/3210779357 learned_addr learned my addr 172.21.15.77:0/3210779357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.025+0000 7f77851c4700 1 --2- 172.21.15.77:0/3210779357 >> v2:172.21.15.159:3300/0 conn(0x7f77800cf440 0x7f778006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7f77859c5700 1 -- 172.21.15.77:0/3210779357 >> v2:172.21.15.159:3300/0 conn(0x7f77800cf440 msgr2=0x7f778006e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7f77859c5700 1 --2- 172.21.15.77:0/3210779357 >> v2:172.21.15.159:3300/0 conn(0x7f77800cf440 0x7f778006e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7f77859c5700 1 -- 172.21.15.77:0/3210779357 >> v1:172.21.15.159:6789/0 conn(0x7f7780070fd0 legacy=0x7f7780165e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7f77859c5700 1 -- 172.21.15.77:0/3210779357 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800717e0 con 0x7f77801665a0 2024-01-15T14:35:11.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7fb328c6a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3241554a0 0x7fb32415b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7fb328c6a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 0x7fb3240b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7fb328c6a700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32406e950 con 0x7fb3241550c0 2024-01-15T14:35:11.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7fb328c6a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32415c220 con 0x7fb3241554a0 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.026+0000 7fb328c6a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32415bc60 con 0x7fb32415b6d0 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 0x7fb3240b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 0x7fb3240b3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53672/0 (socket says 172.21.15.77:53672) 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 -- 172.21.15.77:0/1591925753 learned_addr learned my addr 172.21.15.77:0/1591925753 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb321d9b700 1 --2- 172.21.15.77:0/1591925753 >> v2:172.21.15.77:3300/0 conn(0x7fb3241554a0 0x7fb32415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 -- 172.21.15.77:0/1591925753 >> v2:172.21.15.77:3300/0 conn(0x7fb3241554a0 msgr2=0x7fb32415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 --2- 172.21.15.77:0/1591925753 >> v2:172.21.15.77:3300/0 conn(0x7fb3241554a0 0x7fb32415b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 -- 172.21.15.77:0/1591925753 >> v1:172.21.15.159:6789/0 conn(0x7fb3241550c0 legacy=0x7fb3240ae570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7f25234e3700 1 Processor -- start 2024-01-15T14:35:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb322d9d700 1 -- 172.21.15.77:0/1591925753 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32415bf40 con 0x7fb32415b6d0 2024-01-15T14:35:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7fb321d9b700 1 --2- 172.21.15.77:0/1591925753 >> v2:172.21.15.77:3300/0 conn(0x7fb3241554a0 0x7fb32415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f77859c5700 1 --2- 172.21.15.77:0/3210779357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 0x7f77800b2a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e50 tx=0x7f777800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196afc3517fc620b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.027+0000 7f25234e3700 1 -- start start 2024-01-15T14:35:11.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f25234e3700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f251c0cf440 0x7f251c06e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f25234e3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251c072110 con 0x7f251c070fd0 2024-01-15T14:35:11.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f25234e3700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251c06ee00 con 0x7f251c1665a0 2024-01-15T14:35:11.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f25234e3700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251c071500 con 0x7f251c0cf440 2024-01-15T14:35:11.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f2520a7e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f251c070fd0 0x7f251c165e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38602/0 (socket says 172.21.15.77:38602) 2024-01-15T14:35:11.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f252127f700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f251c0cf440 0x7f251c06e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.028+0000 7f2520a7e700 1 -- 172.21.15.77:0/2818906024 learned_addr learned my addr 172.21.15.77:0/2818906024 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:11.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7fb322d9d700 1 --2- 172.21.15.77:0/1591925753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 0x7fb3240b3bf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009e50 tx=0x7fb31c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa15bbf4c9792dd4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f7777fff700 1 -- 172.21.15.77:0/3210779357 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777800e070 con 0x7f77801665a0 2024-01-15T14:35:11.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f7777fff700 1 -- 172.21.15.77:0/3210779357 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7778003a20 con 0x7f77801665a0 2024-01-15T14:35:11.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f7777fff700 1 -- 172.21.15.77:0/3210779357 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778003dd0 con 0x7f77801665a0 2024-01-15T14:35:11.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f2513fff700 1 -- 172.21.15.77:0/2818906024 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582588297 0 0) 0x7f251c06ee00 con 0x7f251c1665a0 2024-01-15T14:35:11.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f252127f700 1 -- 172.21.15.77:0/2818906024 >> v1:172.21.15.159:6789/0 conn(0x7f251c1665a0 legacy=0x7f251c0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f252127f700 1 -- 172.21.15.77:0/2818906024 >> v1:172.21.15.77:6789/0 conn(0x7f251c070fd0 legacy=0x7f251c165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.029+0000 7f252127f700 1 -- 172.21.15.77:0/2818906024 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c0717e0 con 0x7f251c0cf440 2024-01-15T14:35:11.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7fb32159a700 1 -- 172.21.15.77:0/1591925753 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c00e070 con 0x7fb32415b6d0 2024-01-15T14:35:11.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7fb32159a700 1 -- 172.21.15.77:0/1591925753 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c003a20 con 0x7fb32415b6d0 2024-01-15T14:35:11.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7fb32159a700 1 -- 172.21.15.77:0/1591925753 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c003dd0 con 0x7fb32415b6d0 2024-01-15T14:35:11.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f252127f700 1 --2- 172.21.15.77:0/2818906024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c0cf440 0x7f251c06e8c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2518009840 tx=0x7f2518009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d241d4ee795f7c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f7787428700 1 -- 172.21.15.77:0/3210779357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 msgr2=0x7f77800b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f7787428700 1 --2- 172.21.15.77:0/3210779357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 0x7f77800b2a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e50 tx=0x7f777800b330 comp rx=0 tx=0).stop 2024-01-15T14:35:11.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f7787428700 1 -- 172.21.15.77:0/3210779357 shutdown_connections 2024-01-15T14:35:11.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f7787428700 1 --2- 172.21.15.77:0/3210779357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77801665a0 0x7f77800b2a40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f7787428700 1 --2- 172.21.15.77:0/3210779357 >> v2:172.21.15.159:3300/0 conn(0x7f77800cf440 0x7f778006e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f7787428700 1 -- 172.21.15.77:0/3210779357 >> 172.21.15.77:0/3210779357 conn(0x7f77800b1740 msgr2=0x7f77800b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f2513fff700 1 -- 172.21.15.77:0/2818906024 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2518004230 con 0x7f251c0cf440 2024-01-15T14:35:11.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f2513fff700 1 -- 172.21.15.77:0/2818906024 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2518004390 con 0x7f251c0cf440 2024-01-15T14:35:11.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.030+0000 7f2513fff700 1 -- 172.21.15.77:0/2818906024 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25180036a0 con 0x7f251c0cf440 2024-01-15T14:35:11.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 -- 172.21.15.77:0/1591925753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 msgr2=0x7fb3240b3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 --2- 172.21.15.77:0/1591925753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 0x7fb3240b3bf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009e50 tx=0x7fb31c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:11.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 -- 172.21.15.77:0/2818906024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c0cf440 msgr2=0x7f251c06e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 --2- 172.21.15.77:0/2818906024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c0cf440 0x7f251c06e8c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2518009840 tx=0x7f2518009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:11.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 -- 172.21.15.77:0/2818906024 shutdown_connections 2024-01-15T14:35:11.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 --2- 172.21.15.77:0/2818906024 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c0cf440 0x7f251c06e8c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 -- 172.21.15.77:0/2818906024 >> 172.21.15.77:0/2818906024 conn(0x7f251c0b1740 msgr2=0x7f251c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 -- 172.21.15.77:0/1591925753 shutdown_connections 2024-01-15T14:35:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 --2- 172.21.15.77:0/1591925753 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb32415b6d0 0x7fb3240b3bf0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 --2- 172.21.15.77:0/1591925753 >> v2:172.21.15.77:3300/0 conn(0x7fb3241554a0 0x7fb32415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 -- 172.21.15.77:0/1591925753 >> 172.21.15.77:0/1591925753 conn(0x7fb3240b32a0 msgr2=0x7fb3240d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 -- 172.21.15.77:0/1591925753 shutdown_connections 2024-01-15T14:35:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7fb328c6a700 1 -- 172.21.15.77:0/1591925753 wait complete. 2024-01-15T14:35:11.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 -- 172.21.15.77:0/2818906024 shutdown_connections 2024-01-15T14:35:11.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.031+0000 7f25234e3700 1 -- 172.21.15.77:0/2818906024 wait complete. 2024-01-15T14:35:11.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.032+0000 7f7787428700 1 -- 172.21.15.77:0/3210779357 shutdown_connections 2024-01-15T14:35:11.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.033+0000 7f25234e3700 1 Processor -- start 2024-01-15T14:35:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.033+0000 7fb328c6a700 1 Processor -- start 2024-01-15T14:35:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.033+0000 7fb328c6a700 1 -- start start 2024-01-15T14:35:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7f25234e3700 1 -- start start 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7fb328c6a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3241550c0 0x7fb324153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7fb328c6a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3241554a0 0x7fb3241ef510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7fb328c6a700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32415b6d0 0x7fb3240b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7fb328c6a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324153af0 con 0x7fb3241554a0 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7fb328c6a700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324151830 con 0x7fb32415b6d0 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7fb328c6a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb324151b10 con 0x7fb3241550c0 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3241554a0 0x7fb3241ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3241554a0 0x7fb3241ef510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57392/0 (socket says 172.21.15.77:57392) 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 -- 172.21.15.77:0/2713184292 learned_addr learned my addr 172.21.15.77:0/2713184292 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb32259c700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3241550c0 0x7fb324153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 -- 172.21.15.77:0/2713184292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3241550c0 msgr2=0x7fb324153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb3241550c0 0x7fb324153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 -- 172.21.15.77:0/2713184292 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32415b6d0 msgr2=0x7fb3240b1be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb32415b6d0 0x7fb3240b1be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7fb321d9b700 1 -- 172.21.15.77:0/2713184292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c009b30 con 0x7fb3241554a0 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7fb321d9b700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3241554a0 0x7fb3241ef510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb31400dad0 tx=0x7fb31400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.033+0000 7f7787428700 1 -- 172.21.15.77:0/3210779357 wait complete. 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.034+0000 7f7787428700 1 Processor -- start 2024-01-15T14:35:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f7787428700 1 -- start start 2024-01-15T14:35:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f7787428700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7780070fd0 0x7f77802087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f7787428700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cf440 0x7f7780162bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f7787428700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77801665a0 0x7f77801630f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f7787428700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778020a9b0 con 0x7f77801665a0 2024-01-15T14:35:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f7787428700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780209d30 con 0x7f7780070fd0 2024-01-15T14:35:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f7787428700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778020a080 con 0x7f77800cf440 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f25234e3700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c070fd0 0x7f251c154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f25234e3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f251c0cf440 0x7f251c164a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f25234e3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f251c1665a0 0x7f251c15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f25234e3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c156c10 con 0x7f251c1665a0 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f25234e3700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c1551e0 con 0x7f251c070fd0 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f25234e3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c1653c0 con 0x7f251c0cf440 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f2521a80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f251c1665a0 0x7f251c15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f2521a80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f251c1665a0 0x7f251c15b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57398/0 (socket says 172.21.15.77:57398) 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.035+0000 7f2521a80700 1 -- 172.21.15.77:0/456990572 learned_addr learned my addr 172.21.15.77:0/456990572 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2521a80700 1 -- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f251c0cf440 msgr2=0x7f251c164a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2521a80700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f251c0cf440 0x7f251c164a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2521a80700 1 -- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c070fd0 msgr2=0x7f251c154ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2521a80700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f251c070fd0 0x7f251c154ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2521a80700 1 -- 172.21.15.77:0/456990572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25180094a0 con 0x7f251c1665a0 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2521a80700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f251c1665a0 0x7f251c15b7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f251400c750 tx=0x7f251400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514003c30 con 0x7f251c1665a0 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2514003d90 con 0x7f251c1665a0 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514005290 con 0x7f251c1665a0 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c1562e0 con 0x7f251c1665a0 2024-01-15T14:35:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314004230 con 0x7fb3241554a0 2024-01-15T14:35:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb314004390 con 0x7fb3241554a0 2024-01-15T14:35:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.036+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400a7d0 con 0x7fb3241554a0 2024-01-15T14:35:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.037+0000 7f77859c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77801665a0 0x7f77801630f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.037+0000 7f77859c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77801665a0 0x7f77801630f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57400/0 (socket says 172.21.15.77:57400) 2024-01-15T14:35:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.037+0000 7f77859c5700 1 -- 172.21.15.77:0/3806001619 learned_addr learned my addr 172.21.15.77:0/3806001619 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.037+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c1568a0 con 0x7f251c1665a0 2024-01-15T14:35:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7f77849c3700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cf440 0x7f7780162bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7f77859c5700 1 -- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cf440 msgr2=0x7f7780162bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7f77859c5700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f77800cf440 0x7f7780162bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7f77859c5700 1 -- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7780070fd0 msgr2=0x7f77802087f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7f77859c5700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7780070fd0 0x7f77802087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7f77859c5700 1 -- 172.21.15.77:0/3806001619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778009b30 con 0x7f77801665a0 2024-01-15T14:35:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.039+0000 7f159c941700 1 Processor -- start 2024-01-15T14:35:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb324151df0 con 0x7fb3241554a0 2024-01-15T14:35:11.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.038+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3241514c0 con 0x7fb3241554a0 2024-01-15T14:35:11.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.039+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb32406c2b0 con 0x7fb3241554a0 2024-01-15T14:35:11.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.039+0000 7f77859c5700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77801665a0 0x7f77801630f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7780071640 tx=0x7f7778004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.039+0000 7f159c941700 1 -- start start 2024-01-15T14:35:11.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f24fc000fc0 con 0x7f251c1665a0 2024-01-15T14:35:11.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f159c941700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b4260 0x7f159806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f159c941700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f159806efc0 0x7f1598071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f62f6efe700 1 Processor -- start 2024-01-15T14:35:11.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f159c941700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980714e0 con 0x7f159814be10 2024-01-15T14:35:11.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f159c941700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159814c1f0 con 0x7f159806efc0 2024-01-15T14:35:11.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f159c941700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1598070bf0 con 0x7f15980b4260 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb31400a930 con 0x7fb3241554a0 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7fb312ffd700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb30406b420 0x7fb30406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb314089a50 con 0x7fb3241554a0 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7fb312ffd700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] conn(0x7fb3040735c0 0x7fb3040759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 --> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb304076090 con 0x7fb3040735c0 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7fb322d9d700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] conn(0x7fb3040735c0 0x7fb3040759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7fb32259c700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb30406b420 0x7fb30406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f1595d9b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b4260 0x7f159806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f1595d9b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b4260 0x7f159806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53688/0 (socket says 172.21.15.77:53688) 2024-01-15T14:35:11.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f1595d9b700 1 -- 172.21.15.77:0/461428320 learned_addr learned my addr 172.21.15.77:0/461428320 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25140053f0 con 0x7f251c1665a0 2024-01-15T14:35:11.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f2511ffb700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f250006b420 0x7f250006d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f251409a0f0 con 0x7f251c1665a0 2024-01-15T14:35:11.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f2511ffb700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] conn(0x7f25000735c0 0x7f25000759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 --> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2500076090 con 0x7f25000735c0 2024-01-15T14:35:11.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f1596d9d700 1 --2- 172.21.15.77:0/461428320 >> v2:172.21.15.159:3300/0 conn(0x7f159806efc0 0x7f1598071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7fb322d9d700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] conn(0x7fb3040735c0 0x7fb3040759e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f252127f700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f250006b420 0x7f250006d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7fb32259c700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb30406b420 0x7fb30406d8e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb318006fd0 tx=0x7fb318009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f1596d9d700 1 -- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b4260 msgr2=0x7f159806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f1596d9d700 1 --2- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b4260 0x7f159806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f1596d9d700 1 -- 172.21.15.77:0/461428320 >> v1:172.21.15.77:6789/0 conn(0x7f159814be10 legacy=0x7f15980b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f159559a700 1 -- 172.21.15.77:0/461428320 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498193049 0 0) 0x7f15980714e0 con 0x7f159814be10 2024-01-15T14:35:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f1596d9d700 1 -- 172.21.15.77:0/461428320 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598070e90 con 0x7f159806efc0 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.040+0000 7f62f6efe700 1 -- start start 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f62f6efe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f00afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f62f6efe700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f62f00ae4e0 0x7f62f006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f62f6efe700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f00b46e0 con 0x7f62f006f360 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f62f6efe700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f0166f50 con 0x7f62f00ae4e0 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f62f6efe700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00b49c0 con 0x7f62f0162240 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f62f00ae4e0 0x7f62f006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f62f00ae4e0 0x7f62f006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57792/0 (socket says 172.21.15.77:57792) 2024-01-15T14:35:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 -- 172.21.15.77:0/2163586773 learned_addr learned my addr 172.21.15.77:0/2163586773 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62f4c9a700 1 --2- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f00afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778004440 con 0x7f77801665a0 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77780045a0 con 0x7f77801665a0 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778004930 con 0x7f77801665a0 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778020a360 con 0x7f77801665a0 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7780209000 con 0x7f77801665a0 2024-01-15T14:35:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f252127f700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f250006b420 0x7f250006d8e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f25180090b0 tx=0x7f2518008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.041+0000 7f2520a7e700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] conn(0x7f25000735c0 0x7f25000759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 -- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 msgr2=0x7f62f00afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 --2- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f00afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 -- 172.21.15.77:0/2163586773 >> v1:172.21.15.159:6789/0 conn(0x7f62f006f360 legacy=0x7f62f0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 -- 172.21.15.77:0/2163586773 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f00b4ca0 con 0x7f62f00ae4e0 2024-01-15T14:35:11.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb314054930 con 0x7fb3241554a0 2024-01-15T14:35:11.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f1596d9d700 1 --2- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159806efc0 0x7f1598071970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f158c009840 tx=0x7f158c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4397a5d223b1b0a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f159559a700 1 -- 172.21.15.77:0/461428320 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c011830 con 0x7f159806efc0 2024-01-15T14:35:11.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f159559a700 1 -- 172.21.15.77:0/461428320 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c011990 con 0x7f159806efc0 2024-01-15T14:35:11.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f159559a700 1 -- 172.21.15.77:0/461428320 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c011d20 con 0x7f159806efc0 2024-01-15T14:35:11.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7f62effff700 1 --2- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 0x7f62f006c000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009e50 tx=0x7f62e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d565ba00bd99b2ac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2514064f40 con 0x7f251c1665a0 2024-01-15T14:35:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f159c941700 1 -- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159806efc0 msgr2=0x7f1598071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f159c941700 1 --2- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159806efc0 0x7f1598071970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f158c009840 tx=0x7f158c009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== osd.1 v2:172.21.15.77:6809/1232145209 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb304076090 con 0x7fb3040735c0 2024-01-15T14:35:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f2520a7e700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] conn(0x7f25000735c0 0x7f25000759e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.042+0000 7fa495dba700 1 Processor -- start 2024-01-15T14:35:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7fa495dba700 1 -- start start 2024-01-15T14:35:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f62ef7fe700 1 -- 172.21.15.77:0/2163586773 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e800e070 con 0x7f62f00ae4e0 2024-01-15T14:35:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f62ef7fe700 1 -- 172.21.15.77:0/2163586773 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e8003a20 con 0x7f62f00ae4e0 2024-01-15T14:35:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f62ef7fe700 1 -- 172.21.15.77:0/2163586773 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8003db0 con 0x7f62f00ae4e0 2024-01-15T14:35:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.043+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7760000fc0 con 0x7f77801665a0 2024-01-15T14:35:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f159c941700 1 -- 172.21.15.77:0/461428320 shutdown_connections 2024-01-15T14:35:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f159c941700 1 --2- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159806efc0 0x7f1598071970 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f159c941700 1 --2- 172.21.15.77:0/461428320 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f15980b4260 0x7f159806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f159c941700 1 -- 172.21.15.77:0/461428320 >> 172.21.15.77:0/461428320 conn(0x7f15980cfa30 msgr2=0x7f15980cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa495dba700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa4900b3b50 0x7fa4900b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f62f6efe700 1 -- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 msgr2=0x7f62f006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f62f6efe700 1 --2- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 0x7f62f006c000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009e50 tx=0x7f62e800b330 comp rx=0 tx=0).stop 2024-01-15T14:35:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7778012430 con 0x7f77801665a0 2024-01-15T14:35:11.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== osd.6 v2:172.21.15.159:6808/1865006471 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2500076090 con 0x7f25000735c0 2024-01-15T14:35:11.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f62f6efe700 1 -- 172.21.15.77:0/2163586773 shutdown_connections 2024-01-15T14:35:11.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f62f6efe700 1 --2- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f00afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f62f6efe700 1 --2- 172.21.15.77:0/2163586773 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 0x7f62f006c000 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f62f6efe700 1 -- 172.21.15.77:0/2163586773 >> 172.21.15.77:0/2163586773 conn(0x7f62f0071b90 msgr2=0x7f62f0071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa495dba700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa49006d4d0 0x7fa49006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa495dba700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49006ced0 con 0x7fa490163a80 2024-01-15T14:35:11.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa495dba700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49006da70 con 0x7fa49006d4d0 2024-01-15T14:35:11.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7f159c941700 1 -- 172.21.15.77:0/461428320 shutdown_connections 2024-01-15T14:35:11.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa495dba700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4900b4470 con 0x7fa4900b3b50 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa48ffff700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa490163a80 0x7fa490163ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:35698/0 (socket says 172.21.15.77:35698) 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa48ffff700 1 -- 172.21.15.77:0/3717674154 learned_addr learned my addr 172.21.15.77:0/3717674154 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.044+0000 7fa48f7fe700 1 --2- 172.21.15.77:0/3717674154 >> v2:172.21.15.159:3300/0 conn(0x7fa4900b3b50 0x7fa4900b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48e7fc700 1 -- 172.21.15.77:0/3717674154 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286881796 0 0) 0x7fa49006ced0 con 0x7fa490163a80 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48e7fc700 1 -- 172.21.15.77:0/3717674154 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa47c0035b0 con 0x7fa490163a80 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48effd700 1 --2- 172.21.15.77:0/3717674154 >> v2:172.21.15.77:3300/0 conn(0x7fa49006d4d0 0x7fa49006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3717674154 >> v2:172.21.15.77:3300/0 conn(0x7fa49006d4d0 msgr2=0x7fa49006eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48f7fe700 1 --2- 172.21.15.77:0/3717674154 >> v2:172.21.15.77:3300/0 conn(0x7fa49006d4d0 0x7fa49006eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3717674154 >> v1:172.21.15.159:6789/0 conn(0x7fa490163a80 legacy=0x7fa490163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3717674154 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4901634e0 con 0x7fa4900b3b50 2024-01-15T14:35:11.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f62f6efe700 1 -- 172.21.15.77:0/2163586773 shutdown_connections 2024-01-15T14:35:11.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f7775ffb700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f776406b420 0x7f776406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f777808b100 con 0x7f77801665a0 2024-01-15T14:35:11.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f7775ffb700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] conn(0x7f7764073640 0x7f7764075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 --> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7764076110 con 0x7f7764073640 2024-01-15T14:35:11.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f777808d720 con 0x7f77801665a0 2024-01-15T14:35:11.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f159c941700 1 -- 172.21.15.77:0/461428320 wait complete. 2024-01-15T14:35:11.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7fa48f7fe700 1 --2- 172.21.15.77:0/3717674154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4900b3b50 0x7fa4900b3f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4900750c0 tx=0x7fa478009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=941deb41200aa743 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.046+0000 7fa48e7fc700 1 -- 172.21.15.77:0/3717674154 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478011830 con 0x7fa4900b3b50 2024-01-15T14:35:11.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.046+0000 7fa48e7fc700 1 -- 172.21.15.77:0/3717674154 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478011990 con 0x7fa4900b3b50 2024-01-15T14:35:11.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.046+0000 7fa48e7fc700 1 -- 172.21.15.77:0/3717674154 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478011d20 con 0x7fa4900b3b50 2024-01-15T14:35:11.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.046+0000 7f77849c3700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] conn(0x7f7764073640 0x7f7764075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f159c941700 1 Processor -- start 2024-01-15T14:35:11.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f159c941700 1 -- start start 2024-01-15T14:35:11.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f159c941700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f159806efc0 0x7f15980cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f159c941700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15980b4260 0x7f15980ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f159c941700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159814be10 0x7f15980cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f159c941700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981416d0 con 0x7f15980b4260 2024-01-15T14:35:11.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f159c941700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1598142310 con 0x7f159814be10 2024-01-15T14:35:11.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f159c941700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15981419e0 con 0x7f159806efc0 2024-01-15T14:35:11.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.045+0000 7f62f6efe700 1 -- 172.21.15.77:0/2163586773 wait complete. 2024-01-15T14:35:11.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f62f6efe700 1 Processor -- start 2024-01-15T14:35:11.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f77851c4700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f776406b420 0x7f776406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f77849c3700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] conn(0x7f7764073640 0x7f7764075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f77851c4700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f776406b420 0x7f776406d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f77800af500 tx=0x7f7770009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f1596d9d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159814be10 0x7f15980cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f1596d9d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159814be10 0x7f15980cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57812/0 (socket says 172.21.15.77:57812) 2024-01-15T14:35:11.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f1596d9d700 1 -- 172.21.15.77:0/4063688148 learned_addr learned my addr 172.21.15.77:0/4063688148 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.047+0000 7f62f6efe700 1 -- start start 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f62f6efe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f006f360 0x7f62f0160990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.048+0000 7f62f6efe700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 0x7f62f0139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f6efe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f0139850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f6efe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00b2180 con 0x7f62f006f360 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f6efe700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00b18e0 con 0x7f62f00ae4e0 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f6efe700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00b1bc0 con 0x7f62f0162240 2024-01-15T14:35:11.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f1596d9d700 1 -- 172.21.15.77:0/4063688148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f159806efc0 msgr2=0x7f15980cc840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f1596d9d700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f159806efc0 0x7f15980cc840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f1596d9d700 1 -- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15980b4260 msgr2=0x7f15980ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f1596d9d700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15980b4260 0x7f15980ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f1596d9d700 1 -- 172.21.15.77:0/4063688148 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c0094a0 con 0x7f159814be10 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f1596d9d700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159814be10 0x7f15980cd550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f158c014040 tx=0x7f158c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 -- 172.21.15.77:0/3717674154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4900b3b50 msgr2=0x7fa4900b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 --2- 172.21.15.77:0/3717674154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4900b3b50 0x7fa4900b3f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4900750c0 tx=0x7fa478009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62effff700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 0x7f62f0139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f0139850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f0139850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53708/0 (socket says 172.21.15.77:53708) 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 -- 172.21.15.77:0/3002007492 learned_addr learned my addr 172.21.15.77:0/3002007492 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 -- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 msgr2=0x7f62f0139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62f00ae4e0 0x7f62f0139310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 -- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f006f360 msgr2=0x7f62f0160990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f006f360 0x7f62f0160990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 -- 172.21.15.77:0/3002007492 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e8009b30 con 0x7f62f0162240 2024-01-15T14:35:11.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.049+0000 7f62f549b700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f0139850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c450 tx=0x7f62e400c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== osd.5 v2:172.21.15.159:6819/322902497 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7764076110 con 0x7f7764073640 2024-01-15T14:35:11.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 -- 172.21.15.77:0/3717674154 shutdown_connections 2024-01-15T14:35:11.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 --2- 172.21.15.77:0/3717674154 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4900b3b50 0x7fa4900b3f30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 --2- 172.21.15.77:0/3717674154 >> v2:172.21.15.77:3300/0 conn(0x7fa49006d4d0 0x7fa49006eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 -- 172.21.15.77:0/3717674154 >> 172.21.15.77:0/3717674154 conn(0x7fa4900cfa30 msgr2=0x7fa4900cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 -- 172.21.15.77:0/3717674154 shutdown_connections 2024-01-15T14:35:11.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.050+0000 7fa495dba700 1 -- 172.21.15.77:0/3717674154 wait complete. 2024-01-15T14:35:11.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.051+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c0038a0 con 0x7f159814be10 2024-01-15T14:35:11.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.052+0000 7fa495dba700 1 Processor -- start 2024-01-15T14:35:11.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.051+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c01e070 con 0x7f159814be10 2024-01-15T14:35:11.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.051+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c003c60 con 0x7f159814be10 2024-01-15T14:35:11.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.051+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598141cc0 con 0x7f159814be10 2024-01-15T14:35:11.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.051+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1598143ab0 con 0x7f159814be10 2024-01-15T14:35:11.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.052+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f157c000fc0 con 0x7f159814be10 2024-01-15T14:35:11.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f158c01a420 con 0x7f159814be10 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.052+0000 7fa495dba700 1 -- start start 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7fa495dba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa49006d4d0 0x7fa4900cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7fa495dba700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4900b3b50 0x7fa490156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7fa495dba700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa490163a80 0x7fa490147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7fa495dba700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa490156d60 con 0x7fa49006d4d0 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7fa495dba700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa490156f70 con 0x7fa490163a80 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7fa495dba700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa490157250 con 0x7fa4900b3b50 2024-01-15T14:35:11.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f1586ffd700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f157806b3a0 0x7f157806d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7fa48f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa49006d4d0 0x7fa4900cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7fa48f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa49006d4d0 0x7fa4900cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57416/0 (socket says 172.21.15.77:57416) 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3468696650 learned_addr learned my addr 172.21.15.77:0/3468696650 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3468696650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4900b3b50 msgr2=0x7fa490156820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7fa48f7fe700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4900b3b50 0x7fa490156820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3468696650 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa490163a80 msgr2=0x7fa490147c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7fa48f7fe700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa490163a80 0x7fa490147c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7fa48f7fe700 1 -- 172.21.15.77:0/3468696650 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780094a0 con 0x7fa49006d4d0 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f158c08a3b0 con 0x7f159814be10 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f1586ffd700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] conn(0x7f1578073540 0x7f1578075960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 --> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1578076010 con 0x7f1578073540 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f1595d9b700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] conn(0x7f1578073540 0x7f1578075960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f159659c700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f157806b3a0 0x7f157806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7f159659c700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f157806b3a0 0x7f157806d860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1590001d40 tx=0x7f1590007cc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.055+0000 7f1595d9b700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] conn(0x7f1578073540 0x7f1578075960 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7fa48f7fe700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa49006d4d0 0x7fa4900cf1b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa478009b60 tx=0x7fa478004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4780036a0 con 0x7fa49006d4d0 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47801f070 con 0x7fa49006d4d0 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478003a40 con 0x7fa49006d4d0 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4011790 con 0x7f62f0162240 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e4011910 con 0x7f62f0162240 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.053+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4011cc0 con 0x7f62f0162240 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f00b1ea0 con 0x7f62f0162240 2024-01-15T14:35:11.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.054+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f015f830 con 0x7f62f0162240 2024-01-15T14:35:11.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f158c0552e0 con 0x7f159814be10 2024-01-15T14:35:11.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.057+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== osd.2 v2:172.21.15.77:6800/1734185656 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1578076010 con 0x7f1578073540 2024-01-15T14:35:11.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa490157530 con 0x7fa49006d4d0 2024-01-15T14:35:11.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa490157af0 con 0x7fa49006d4d0 2024-01-15T14:35:11.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.057+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa490165bf0 con 0x7fa49006d4d0 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.056+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f62e401c460 con 0x7f62f0162240 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.057+0000 7f62ed7fa700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62dc06b420 0x7f62dc06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.057+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62e40898b0 con 0x7f62f0162240 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.058+0000 7f92ba697700 1 Processor -- start 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.058+0000 7f92ba697700 1 -- start start 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92ba697700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92b40b3b50 0x7f92b40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92ba697700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92b406d4d0 0x7f92b406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92ba697700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b406ced0 con 0x7f92b4163a80 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92ba697700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b406da70 con 0x7f92b40b3b50 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92ba697700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b40b4470 con 0x7f92b406d4d0 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92b8c34700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f92b4163a80 0x7f92b4163ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38616/0 (socket says 172.21.15.77:38616) 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.059+0000 7f92b8c34700 1 -- 172.21.15.77:0/1458327183 learned_addr learned my addr 172.21.15.77:0/1458327183 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7f92b37fe700 1 --2- 172.21.15.77:0/1458327183 >> v2:172.21.15.159:3300/0 conn(0x7f92b406d4d0 0x7f92b406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa478003ba0 con 0x7fa49006d4d0 2024-01-15T14:35:11.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7fa487fff700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa47c06b870 0x7fa47c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7f62f4c9a700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62dc06b420 0x7f62dc06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7f92b37fe700 1 -- 172.21.15.77:0/1458327183 >> v2:172.21.15.77:3300/0 conn(0x7f92b40b3b50 msgr2=0x7f92b40b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7f92b37fe700 1 --2- 172.21.15.77:0/1458327183 >> v2:172.21.15.77:3300/0 conn(0x7f92b40b3b50 0x7f92b40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7f92b37fe700 1 -- 172.21.15.77:0/1458327183 >> v1:172.21.15.77:6789/0 conn(0x7f92b4163a80 legacy=0x7f92b4163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7f92b37fe700 1 -- 172.21.15.77:0/1458327183 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b41634e0 con 0x7f92b406d4d0 2024-01-15T14:35:11.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa478021070 con 0x7fa49006d4d0 2024-01-15T14:35:11.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.061+0000 7f92b37fe700 1 --2- 172.21.15.77:0/1458327183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b406d4d0 0x7f92b406eee0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f92b4062350 tx=0x7f92a4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd149bd0f3b470ad server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7fa487fff700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] conn(0x7fa47c073a90 0x7fa47c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 --> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa47c076560 con 0x7fa47c073a90 2024-01-15T14:35:11.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.060+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa47808bd20 con 0x7fa49006d4d0 2024-01-15T14:35:11.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.061+0000 7fa48ffff700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] conn(0x7fa47c073a90 0x7fa47c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.061+0000 7fa48effd700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa47c06b870 0x7fa47c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.061+0000 7f62f4c9a700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62dc06b420 0x7f62dc06d8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62e00099d0 tx=0x7f62e0009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.061+0000 7f62f6efe700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] conn(0x7f62f015fab0 0x7f62f015fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.061+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 --> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62f00b5260 con 0x7f62f015fab0 2024-01-15T14:35:11.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.062+0000 7fa48effd700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa47c06b870 0x7fa47c06dd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa490154c20 tx=0x7fa480009790 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.062+0000 7fa48ffff700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] conn(0x7fa47c073a90 0x7fa47c075eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.062+0000 7f92b2ffd700 1 -- 172.21.15.77:0/1458327183 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4004230 con 0x7f92b406d4d0 2024-01-15T14:35:11.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.062+0000 7f92b2ffd700 1 -- 172.21.15.77:0/1458327183 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a4004390 con 0x7f92b406d4d0 2024-01-15T14:35:11.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.062+0000 7f92b2ffd700 1 -- 172.21.15.77:0/1458327183 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a40036a0 con 0x7f92b406d4d0 2024-01-15T14:35:11.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f62effff700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] conn(0x7f62f015fab0 0x7f62f015fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 -- 172.21.15.77:0/1458327183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b406d4d0 msgr2=0x7f92b406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 --2- 172.21.15.77:0/1458327183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b406d4d0 0x7f92b406eee0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f92b4062350 tx=0x7f92a4009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 -- 172.21.15.77:0/1458327183 shutdown_connections 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 --2- 172.21.15.77:0/1458327183 >> v2:172.21.15.77:3300/0 conn(0x7f92b40b3b50 0x7f92b40b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 --2- 172.21.15.77:0/1458327183 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b406d4d0 0x7f92b406eee0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 -- 172.21.15.77:0/1458327183 >> 172.21.15.77:0/1458327183 conn(0x7f92b40cfa30 msgr2=0x7f92b40cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f62effff700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] conn(0x7f62f015fab0 0x7f62f015fe90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 -- 172.21.15.77:0/1458327183 shutdown_connections 2024-01-15T14:35:11.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7f92ba697700 1 -- 172.21.15.77:0/1458327183 wait complete. 2024-01-15T14:35:11.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.063+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== osd.3 v2:172.21.15.77:6808/3087735168 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa47c076560 con 0x7fa47c073a90 2024-01-15T14:35:11.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.064+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== osd.4 v2:172.21.15.159:6800/3726410932 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f62e80092e0 con 0x7f62f015fab0 2024-01-15T14:35:11.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.064+0000 7f92ba697700 1 Processor -- start 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.065+0000 7f92ba697700 1 -- start start 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92ba697700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92b406d4d0 0x7f92b4140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92ba697700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b40b3b50 0x7f92b4142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92ba697700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b4163a80 0x7f92b4148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92ba697700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4147420 con 0x7f92b40b3b50 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92ba697700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4141210 con 0x7f92b4163a80 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92ba697700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4146e60 con 0x7f92b406d4d0 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92b3fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92b406d4d0 0x7f92b4140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92b3fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92b406d4d0 0x7f92b4140cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53724/0 (socket says 172.21.15.77:53724) 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92b3fff700 1 -- 172.21.15.77:0/1427715087 learned_addr learned my addr 172.21.15.77:0/1427715087 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92b8c34700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b4163a80 0x7f92b4148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.066+0000 7f92b37fe700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b40b3b50 0x7f92b4142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f92b3fff700 1 -- 172.21.15.77:0/1427715087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b4163a80 msgr2=0x7f92b4148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f92b3fff700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92b4163a80 0x7f92b4148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f92b3fff700 1 -- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b40b3b50 msgr2=0x7f92b4142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f92b3fff700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92b40b3b50 0x7f92b4142000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f92b3fff700 1 -- 172.21.15.77:0/1427715087 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40094a0 con 0x7f92b406d4d0 2024-01-15T14:35:11.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f21924dc700 1 Processor -- start 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.067+0000 7f21924dc700 1 -- start start 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f21924dc700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f218c0b3370 0x7f218c1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f21924dc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c0b22d0 con 0x7f218c0d1c70 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f21924dc700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c0b25b0 con 0x7f218c15bdd0 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f21924dc700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0b2f50 con 0x7f218c0b3370 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218bfff700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f218c15bdd0 0x7f218c0b1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:35714/0 (socket says 172.21.15.77:35714) 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218bfff700 1 -- 172.21.15.77:0/2498585379 learned_addr learned my addr 172.21.15.77:0/2498585379 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218b7fe700 1 --2- 172.21.15.77:0/2498585379 >> v2:172.21.15.159:3300/0 conn(0x7f218c0b3370 0x7f218c1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564507628 0 0) 0x7f218c0b25b0 con 0x7f218c15bdd0 2024-01-15T14:35:11.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21740035b0 con 0x7f218c15bdd0 2024-01-15T14:35:11.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f92b3fff700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92b406d4d0 0x7f92b4140cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b530 tx=0x7f92a800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3096608722 0 0) 0x7f21740035b0 con 0x7f218c15bdd0 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 >> v2:172.21.15.159:3300/0 conn(0x7f218c0b3370 msgr2=0x7f218c1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 --2- 172.21.15.77:0/2498585379 >> v2:172.21.15.159:3300/0 conn(0x7f218c0b3370 0x7f218c1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 >> v1:172.21.15.77:6789/0 conn(0x7f218c0d1c70 legacy=0x7f218c165ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0b2890 con 0x7f218c15bdd0 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f217c001670 con 0x7f218c15bdd0 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 >> v1:172.21.15.159:6789/0 conn(0x7f218c15bdd0 legacy=0x7f218c0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2174003a00 0x7f2174003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2174004850 0x7f2174004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174005210 0x7f217400fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b2f50 con 0x7f2174004850 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174010550 con 0x7f2174005210 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.068+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2174010710 con 0x7f2174003a00 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2174003a00 0x7f2174003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f2190a79700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174005210 0x7f217400fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 -- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174005210 msgr2=0x7f217400fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174005210 0x7f217400fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 -- 172.21.15.77:0/2498585379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2174004850 msgr2=0x7f2174004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2174004850 0x7f2174004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 -- 172.21.15.77:0/2498585379 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0b2890 con 0x7f2174003a00 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f2190a79700 1 -- 172.21.15.77:0/2498585379 reap_dead start 2024-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218bfff700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2174003a00 0x7f2174003e00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f217c00c340 tx=0x7f2174005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b950073a04d7900a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c00ce80 con 0x7f2174003a00 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c005960 con 0x7f2174003a00 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.069+0000 7f218affd700 1 -- 172.21.15.77:0/2498585379 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c005cf0 con 0x7f2174003a00 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.070+0000 7f21924dc700 1 -- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2174003a00 msgr2=0x7f2174003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.070+0000 7f21924dc700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2174003a00 0x7f2174003e00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f217c00c340 tx=0x7f2174005880 comp rx=0 tx=0).stop 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.071+0000 7f21924dc700 1 -- 172.21.15.77:0/2498585379 shutdown_connections 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.071+0000 7f21924dc700 1 --2- 172.21.15.77:0/2498585379 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2174003a00 0x7f2174003e00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.071+0000 7f21924dc700 1 -- 172.21.15.77:0/2498585379 >> 172.21.15.77:0/2498585379 conn(0x7f218c0731a0 msgr2=0x7f218c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.071+0000 7f21924dc700 1 -- 172.21.15.77:0/2498585379 shutdown_connections 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.071+0000 7f21924dc700 1 -- 172.21.15.77:0/2498585379 wait complete. 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.073+0000 7f21924dc700 1 Processor -- start 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.073+0000 7f21924dc700 1 -- start start 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f21924dc700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174003a00 0x7f218c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f21924dc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c0cc680 0x7f218c0b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f21924dc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c0b0730 0x7f218c0ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f21924dc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b02a0 con 0x7f218c0b0730 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f21924dc700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0ccb80 con 0x7f2174003a00 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f21924dc700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0af970 con 0x7f218c0cc680 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f218bfff700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174003a00 0x7f218c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f218bfff700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174003a00 0x7f218c0b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57860/0 (socket says 172.21.15.77:57860) 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.074+0000 7f218bfff700 1 -- 172.21.15.77:0/138036485 learned_addr learned my addr 172.21.15.77:0/138036485 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f2190a79700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c0b0730 0x7f218c0ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 --> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f24fc002d40 con 0x7f25000735c0 2024-01-15T14:35:11.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8011830 con 0x7f92b406d4d0 2024-01-15T14:35:11.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 --> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb32406db90 con 0x7fb3040735c0 2024-01-15T14:35:11.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218b7fe700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c0cc680 0x7f218c0b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a80119b0 con 0x7f92b406d4d0 2024-01-15T14:35:11.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8011d60 con 0x7f92b406d4d0 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218bfff700 1 -- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c0cc680 msgr2=0x7f218c0b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218bfff700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c0cc680 0x7f218c0b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218bfff700 1 -- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c0b0730 msgr2=0x7f218c0ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218bfff700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c0b0730 0x7f218c0ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218bfff700 1 -- 172.21.15.77:0/138036485 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0afc50 con 0x7f2174003a00 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b4147140 con 0x7f92b406d4d0 2024-01-15T14:35:11.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b4141800 con 0x7f92b406d4d0 2024-01-15T14:35:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f2190a79700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c0b0730 0x7f218c0ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218b7fe700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f218c0cc680 0x7f218c0b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f218bfff700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174003a00 0x7f218c0b4f90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21740108a0 tx=0x7f217c001930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f2511ffb700 1 -- 172.21.15.77:0/456990572 <== osd.6 v2:172.21.15.159:6808/1865006471 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f24fc002d40 con 0x7f25000735c0 2024-01-15T14:35:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.075+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c001bf0 con 0x7f2174003a00 2024-01-15T14:35:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c0015e0 con 0x7f2174003a00 2024-01-15T14:35:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c004ca0 con 0x7f2174003a00 2024-01-15T14:35:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c0aff30 con 0x7f2174003a00 2024-01-15T14:35:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c0b5a10 con 0x7f2174003a00 2024-01-15T14:35:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7fb312ffd700 1 -- 172.21.15.77:0/2713184292 <== osd.1 v2:172.21.15.77:6809/1232145209 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb32406db90 con 0x7fb3040735c0 2024-01-15T14:35:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] conn(0x7f25000735c0 msgr2=0x7f25000759e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.076+0000 7f25234e3700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.159:6808/1865006471,v1:172.21.15.159:6809/1865006471] conn(0x7f25000735c0 0x7f25000759e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.077+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f218c1604e0 con 0x7f2174003a00 2024-01-15T14:35:11.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.077+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f92a801a400 con 0x7f92b406d4d0 2024-01-15T14:35:11.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7f92b0ff9700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92a006b3a0 0x7f92a006d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92a8089b20 con 0x7f92b406d4d0 2024-01-15T14:35:11.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] conn(0x7fb3040735c0 msgr2=0x7fb3040759e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7fb328c6a700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6809/1232145209,v1:172.21.15.77:6811/1232145209] conn(0x7fb3040735c0 0x7fb3040759e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb30406b420 msgr2=0x7fb30406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7fb328c6a700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb30406b420 0x7fb30406d8e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb318006fd0 tx=0x7fb318009190 comp rx=0 tx=0).stop 2024-01-15T14:35:11.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3241554a0 msgr2=0x7fb3241ef510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7fb328c6a700 1 --2- 172.21.15.77:0/2713184292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3241554a0 0x7fb3241ef510 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb31400dad0 tx=0x7fb31400dde0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f217c0070d0 con 0x7f2174003a00 2024-01-15T14:35:11.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7f2188ff9700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f217806b420 0x7f217806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.078+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f217c08b0a0 con 0x7f2174003a00 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f2188ff9700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] conn(0x7f2178073640 0x7f2178075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 --> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2178076110 con 0x7f2178073640 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f217c08b450 con 0x7f2174003a00 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f2190a79700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] conn(0x7f2178073640 0x7f2178075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f2190a79700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] conn(0x7f2178073640 0x7f2178075a60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f92b37fe700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92a006b3a0 0x7f92a006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.079+0000 7f92b37fe700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92a006b3a0 0x7f92a006d860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f92b4062350 tx=0x7f92a4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.077+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f250006b420 msgr2=0x7f250006d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.077+0000 7f25234e3700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f250006b420 0x7f250006d8e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f25180090b0 tx=0x7f2518008030 comp rx=0 tx=0).stop 2024-01-15T14:35:11.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.077+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f251c1665a0 msgr2=0x7f251c15b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.077+0000 7f25234e3700 1 --2- 172.21.15.77:0/456990572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f251c1665a0 0x7f251c15b7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f251400c750 tx=0x7f251400cb10 comp rx=0 tx=0).stop 2024-01-15T14:35:11.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== osd.7 v2:172.21.15.159:6816/4215207465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2178076110 con 0x7f2178073640 2024-01-15T14:35:11.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7fb322d9d700 1 -- 172.21.15.77:0/2713184292 reap_dead start 2024-01-15T14:35:11.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 shutdown_connections 2024-01-15T14:35:11.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 >> 172.21.15.77:0/2713184292 conn(0x7fb3240b32a0 msgr2=0x7fb3240ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7f92ba697700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] conn(0x7f92940015e0 0x7f9294003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 --> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9294006bd0 con 0x7f92940015e0 2024-01-15T14:35:11.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.080+0000 7f92b8c34700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] conn(0x7f92940015e0 0x7f9294003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.082+0000 7f92b8c34700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] conn(0x7f92940015e0 0x7f9294003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.081+0000 7f2521a80700 1 -- 172.21.15.77:0/456990572 reap_dead start 2024-01-15T14:35:11.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.082+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 shutdown_connections 2024-01-15T14:35:11.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.082+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 >> 172.21.15.77:0/456990572 conn(0x7f251c0b1740 msgr2=0x7f251c162730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.083+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 shutdown_connections 2024-01-15T14:35:11.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.083+0000 7fb328c6a700 1 -- 172.21.15.77:0/2713184292 wait complete. 2024-01-15T14:35:11.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.083+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== osd.0 v2:172.21.15.77:6824/2427925803 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9294006bd0 con 0x7f92940015e0 2024-01-15T14:35:11.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.083+0000 7f218b7fe700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f217806b420 0x7f217806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.084+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 shutdown_connections 2024-01-15T14:35:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.084+0000 7f218b7fe700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f217806b420 0x7f217806d8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f218c0d13e0 tx=0x7f2180008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.084+0000 7f25234e3700 1 -- 172.21.15.77:0/456990572 wait complete. 2024-01-15T14:35:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.091+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 --> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa49006cb60 con 0x7fa47c073a90 2024-01-15T14:35:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.091+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 --> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f157c002d40 con 0x7f1578073540 2024-01-15T14:35:11.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.092+0000 7fa487fff700 1 -- 172.21.15.77:0/3468696650 <== osd.3 v2:172.21.15.77:6808/3087735168 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa488007ec0 con 0x7fa47c073a90 2024-01-15T14:35:11.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.093+0000 7f1586ffd700 1 -- 172.21.15.77:0/4063688148 <== osd.2 v2:172.21.15.77:6800/1734185656 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f157c002d40 con 0x7f1578073540 2024-01-15T14:35:11.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.094+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] conn(0x7f1578073540 msgr2=0x7f1578075960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.094+0000 7f159c941700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6800/1734185656,v1:172.21.15.77:6801/1734185656] conn(0x7f1578073540 0x7f1578075960 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f157806b3a0 msgr2=0x7f157806d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f159c941700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f157806b3a0 0x7f157806d860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1590001d40 tx=0x7f1590007cc0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159814be10 msgr2=0x7f15980cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f159c941700 1 --2- 172.21.15.77:0/4063688148 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f159814be10 0x7f15980cd550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f158c014040 tx=0x7f158c004750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f1596d9d700 1 -- 172.21.15.77:0/4063688148 reap_dead start 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 --> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7760002d40 con 0x7f7764073640 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 shutdown_connections 2024-01-15T14:35:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.095+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 >> 172.21.15.77:0/4063688148 conn(0x7f15980cfa30 msgr2=0x7f159814af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] conn(0x7fa47c073a90 msgr2=0x7fa47c075eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa495dba700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6808/3087735168,v1:172.21.15.77:6810/3087735168] conn(0x7fa47c073a90 0x7fa47c075eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 shutdown_connections 2024-01-15T14:35:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7f7775ffb700 1 -- 172.21.15.77:0/3806001619 <== osd.5 v2:172.21.15.159:6819/322902497 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7760002d40 con 0x7f7764073640 2024-01-15T14:35:11.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7f159c941700 1 -- 172.21.15.77:0/4063688148 wait complete. 2024-01-15T14:35:11.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa47c06b870 msgr2=0x7fa47c06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa495dba700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa47c06b870 0x7fa47c06dd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa490154c20 tx=0x7fa480009790 comp rx=0 tx=0).stop 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa49006d4d0 msgr2=0x7fa4900cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa495dba700 1 --2- 172.21.15.77:0/3468696650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa49006d4d0 0x7fa4900cf1b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa478009b60 tx=0x7fa478004750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.096+0000 7fa48ffff700 1 -- 172.21.15.77:0/3468696650 reap_dead start 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.097+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 shutdown_connections 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.097+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 >> 172.21.15.77:0/3468696650 conn(0x7fa4900cfa30 msgr2=0x7fa4900af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.097+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 shutdown_connections 2024-01-15T14:35:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.097+0000 7fa495dba700 1 -- 172.21.15.77:0/3468696650 wait complete. 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.099+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] conn(0x7f7764073640 msgr2=0x7f7764075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.099+0000 7f7787428700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.159:6819/322902497,v1:172.21.15.159:6821/322902497] conn(0x7f7764073640 0x7f7764075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.099+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f776406b420 msgr2=0x7f776406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.099+0000 7f7787428700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f776406b420 0x7f776406d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f77800af500 tx=0x7f7770009210 comp rx=0 tx=0).stop 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.099+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77801665a0 msgr2=0x7f77801630f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.099+0000 7f7787428700 1 --2- 172.21.15.77:0/3806001619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77801665a0 0x7f77801630f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7780071640 tx=0x7f7778004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.100+0000 7f77859c5700 1 -- 172.21.15.77:0/3806001619 reap_dead start 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.103+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 shutdown_connections 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.103+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 >> 172.21.15.77:0/3806001619 conn(0x7f77800b1740 msgr2=0x7f77800b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.104+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 --> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f218c067f60 con 0x7f2178073640 2024-01-15T14:35:11.121 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:11.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T14:35:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.107+0000 7f2188ff9700 1 -- 172.21.15.77:0/138036485 <== osd.7 v2:172.21.15.159:6816/4215207465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f218c067f60 con 0x7f2178073640 2024-01-15T14:35:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.107+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] conn(0x7f2178073640 msgr2=0x7f2178075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.107+0000 7f21924dc700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:6816/4215207465,v1:172.21.15.159:6817/4215207465] conn(0x7f2178073640 0x7f2178075a60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.108+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 shutdown_connections 2024-01-15T14:35:11.123 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:11.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T14:35:11.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.110+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 --> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62f0162620 con 0x7f62f015fab0 2024-01-15T14:35:11.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.110+0000 7f62ed7fa700 1 -- 172.21.15.77:0/3002007492 <== osd.4 v2:172.21.15.159:6800/3726410932 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f62e800b8d0 con 0x7f62f015fab0 2024-01-15T14:35:11.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] conn(0x7f62f015fab0 msgr2=0x7f62f015fe90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:6800/3726410932,v1:172.21.15.159:6801/3726410932] conn(0x7f62f015fab0 0x7f62f015fe90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62dc06b420 msgr2=0x7f62dc06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62dc06b420 0x7f62dc06d8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62e00099d0 tx=0x7f62e0009210 comp rx=0 tx=0).stop 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 msgr2=0x7f62f0139850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 --2- 172.21.15.77:0/3002007492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62f0162240 0x7f62f0139850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c450 tx=0x7f62e400c760 comp rx=0 tx=0).stop 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f549b700 1 -- 172.21.15.77:0/3002007492 reap_dead start 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 shutdown_connections 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 >> 172.21.15.77:0/3002007492 conn(0x7f62f0071b90 msgr2=0x7f62f006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 shutdown_connections 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f62f6efe700 1 -- 172.21.15.77:0/3002007492 wait complete. 2024-01-15T14:35:11.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f7787428700 1 -- 172.21.15.77:0/3806001619 wait complete. 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.111+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 --> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9294005cb0 con 0x7f92940015e0 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f217806b420 msgr2=0x7f217806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f217806b420 0x7f217806d8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f218c0d13e0 tx=0x7f2180008040 comp rx=0 tx=0).stop 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174003a00 msgr2=0x7f218c0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 --2- 172.21.15.77:0/138036485 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2174003a00 0x7f218c0b4f90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21740108a0 tx=0x7f217c001930 comp rx=0 tx=0).stop 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f2190a79700 1 -- 172.21.15.77:0/138036485 reap_dead start 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 shutdown_connections 2024-01-15T14:35:11.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 >> 172.21.15.77:0/138036485 conn(0x7f218c0731a0 msgr2=0x7f218c157d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f92b0ff9700 1 -- 172.21.15.77:0/1427715087 <== osd.0 v2:172.21.15.77:6824/2427925803 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9294005cb0 con 0x7f92940015e0 2024-01-15T14:35:11.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 shutdown_connections 2024-01-15T14:35:11.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f21924dc700 1 -- 172.21.15.77:0/138036485 wait complete. 2024-01-15T14:35:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] conn(0x7f92940015e0 msgr2=0x7f9294003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.112+0000 7f92ba697700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6824/2427925803,v1:172.21.15.77:6825/2427925803] conn(0x7f92940015e0 0x7f9294003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92a006b3a0 msgr2=0x7f92a006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92a006b3a0 0x7f92a006d860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f92b4062350 tx=0x7f92a4008030 comp rx=0 tx=0).stop 2024-01-15T14:35:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92b406d4d0 msgr2=0x7f92b4140cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 --2- 172.21.15.77:0/1427715087 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92b406d4d0 0x7f92b4140cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b530 tx=0x7f92a800b840 comp rx=0 tx=0).stop 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92b8c34700 1 -- 172.21.15.77:0/1427715087 reap_dead start 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 shutdown_connections 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 >> 172.21.15.77:0/1427715087 conn(0x7f92b40cfa30 msgr2=0x7f92b4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 shutdown_connections 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.113+0000 7f92ba697700 1 -- 172.21.15.77:0/1427715087 wait complete. 2024-01-15T14:35:11.130 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T14:35:11.131 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T14:35:11.133 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T14:35:11.134 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:11.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T14:35:11.135 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T14:35:11.137 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:11.137 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T14:35:11.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.324+0000 7faeed7ff700 1 Processor -- start 2024-01-15T14:35:11.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.324+0000 7faeed7ff700 1 -- start start 2024-01-15T14:35:11.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.325+0000 7faeed7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faee80d0520 0x7faee815d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.325+0000 7faeed7ff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80b3b50 0x7faee80b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.325+0000 7faeed7ff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faee80714d0 0x7faee8071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faeed7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee8072d60 con 0x7faee80d0520 2024-01-15T14:35:11.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faeed7ff700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee806de80 con 0x7faee80714d0 2024-01-15T14:35:11.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faeed7ff700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80b4510 con 0x7faee80b3b50 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee6ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faee80d0520 0x7faee815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee6ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faee80d0520 0x7faee815d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57466/0 (socket says 172.21.15.77:57466) 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee6ffd700 1 -- 172.21.15.77:0/4116667812 learned_addr learned my addr 172.21.15.77:0/4116667812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee67fc700 1 --2- 172.21.15.77:0/4116667812 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80b3b50 0x7faee80b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee77fe700 1 --2- 172.21.15.77:0/4116667812 >> v2:172.21.15.159:3300/0 conn(0x7faee80714d0 0x7faee8071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee67fc700 1 -- 172.21.15.77:0/4116667812 >> v2:172.21.15.159:3300/0 conn(0x7faee80714d0 msgr2=0x7faee8071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee67fc700 1 --2- 172.21.15.77:0/4116667812 >> v2:172.21.15.159:3300/0 conn(0x7faee80714d0 0x7faee8071950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee67fc700 1 -- 172.21.15.77:0/4116667812 >> v2:172.21.15.77:3300/0 conn(0x7faee80d0520 msgr2=0x7faee815d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.326+0000 7faee67fc700 1 --2- 172.21.15.77:0/4116667812 >> v2:172.21.15.77:3300/0 conn(0x7faee80d0520 0x7faee815d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.327+0000 7faee67fc700 1 -- 172.21.15.77:0/4116667812 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee815d970 con 0x7faee80b3b50 2024-01-15T14:35:11.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.327+0000 7faee67fc700 1 --2- 172.21.15.77:0/4116667812 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80b3b50 0x7faee80b3fd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faee80750f0 tx=0x7faedc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a697da97dcd626c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.327+0000 7f1153caf700 1 Processor -- start 2024-01-15T14:35:11.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.327+0000 7f1153caf700 1 -- start start 2024-01-15T14:35:11.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.328+0000 7faee5ffb700 1 -- 172.21.15.77:0/4116667812 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc011830 con 0x7faee80b3b50 2024-01-15T14:35:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.328+0000 7faee5ffb700 1 -- 172.21.15.77:0/4116667812 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc011990 con 0x7faee80b3b50 2024-01-15T14:35:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.328+0000 7faee5ffb700 1 -- 172.21.15.77:0/4116667812 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc011d20 con 0x7faee80b3b50 2024-01-15T14:35:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7faeed7ff700 1 -- 172.21.15.77:0/4116667812 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80b3b50 msgr2=0x7faee80b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7faeed7ff700 1 --2- 172.21.15.77:0/4116667812 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80b3b50 0x7faee80b3fd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faee80750f0 tx=0x7faedc009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f1153caf700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f114c164a90 0x7f114c06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f1153caf700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c06d4d0 0x7f114c16bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f1153caf700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c06ced0 con 0x7f114c0b3b50 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f1153caf700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c06da70 con 0x7f114c164a90 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f1153caf700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c1634e0 con 0x7f114c06d4d0 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f115124a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f114c164a90 0x7f114c06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f115124a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f114c164a90 0x7f114c06eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57470/0 (socket says 172.21.15.77:57470) 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f115124a700 1 -- 172.21.15.77:0/1160069505 learned_addr learned my addr 172.21.15.77:0/1160069505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f115124a700 1 -- 172.21.15.77:0/1160069505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c06d4d0 msgr2=0x7f114c16bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.329+0000 7f115124a700 1 --2- 172.21.15.77:0/1160069505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c06d4d0 0x7f114c16bee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7f115124a700 1 -- 172.21.15.77:0/1160069505 >> v1:172.21.15.159:6789/0 conn(0x7f114c0b3b50 legacy=0x7f114c0b3f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7f115124a700 1 -- 172.21.15.77:0/1160069505 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c1637c0 con 0x7f114c164a90 2024-01-15T14:35:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7f115124a700 1 --2- 172.21.15.77:0/1160069505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c164a90 0x7f114c06eee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1144009e50 tx=0x7f114400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bd44adecdb3d46 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7faeed7ff700 1 -- 172.21.15.77:0/4116667812 shutdown_connections 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7faeed7ff700 1 --2- 172.21.15.77:0/4116667812 >> v2:172.21.15.159:3300/0 conn(0x7faee80714d0 0x7faee8071950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7faeed7ff700 1 --2- 172.21.15.77:0/4116667812 >> v2:172.21.15.77:3300/0 conn(0x7faee80d0520 0x7faee815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7faeed7ff700 1 --2- 172.21.15.77:0/4116667812 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80b3b50 0x7faee80b3fd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7faeed7ff700 1 -- 172.21.15.77:0/4116667812 >> 172.21.15.77:0/4116667812 conn(0x7faee80cfa30 msgr2=0x7faee80cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.330+0000 7faeed7ff700 1 -- 172.21.15.77:0/4116667812 shutdown_connections 2024-01-15T14:35:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.331+0000 7faeed7ff700 1 -- 172.21.15.77:0/4116667812 wait complete. 2024-01-15T14:35:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.333+0000 7faeed7ff700 1 Processor -- start 2024-01-15T14:35:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.333+0000 7faeed7ff700 1 -- start start 2024-01-15T14:35:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faeed7ff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 0x7faee814d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faeed7ff700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faee80b3b50 0x7faee8136af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faeed7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faee80d0520 0x7faee8152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faeed7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee8153690 con 0x7faee80d0520 2024-01-15T14:35:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faeed7ff700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81406c0 con 0x7faee80b3b50 2024-01-15T14:35:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faeed7ff700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee8140420 con 0x7faee80714d0 2024-01-15T14:35:11.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.334+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 0x7faee814d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 0x7faee814d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53760/0 (socket says 172.21.15.77:53760) 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee6ffd700 1 -- 172.21.15.77:0/39864631 learned_addr learned my addr 172.21.15.77:0/39864631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee67fc700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faee80b3b50 0x7faee8136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee77fe700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faee80d0520 0x7faee8152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee67fc700 1 -- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 msgr2=0x7faee814d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee67fc700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 0x7faee814d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee67fc700 1 -- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faee80d0520 msgr2=0x7faee8152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee67fc700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faee80d0520 0x7faee8152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee67fc700 1 -- 172.21.15.77:0/39864631 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0094a0 con 0x7faee80b3b50 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee77fe700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faee80d0520 0x7faee8152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.335+0000 7faee6ffd700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 0x7faee814d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.336+0000 7faee67fc700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faee80b3b50 0x7faee8136af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faedc016000 tx=0x7faedc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.336+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc0036a0 con 0x7faee80b3b50 2024-01-15T14:35:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.336+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc01f070 con 0x7faee80b3b50 2024-01-15T14:35:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.336+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc003a00 con 0x7faee80b3b50 2024-01-15T14:35:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.336+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee8139bb0 con 0x7faee80b3b50 2024-01-15T14:35:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.336+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee814f790 con 0x7faee80b3b50 2024-01-15T14:35:11.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.339+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faedc003b60 con 0x7faee80b3b50 2024-01-15T14:35:11.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.339+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec80052f0 con 0x7faee80b3b50 2024-01-15T14:35:11.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.339+0000 7faecf7fe700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7faed406b420 0x7faed406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.339+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faedc021070 con 0x7faee80b3b50 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.340+0000 7faee6ffd700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7faed406b420 0x7faed406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.340+0000 7faee6ffd700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7faed406b420 0x7faed406d8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faed8006b20 tx=0x7faed8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.340+0000 7f1150a49700 1 -- 172.21.15.77:0/1160069505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114400e070 con 0x7f114c164a90 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.340+0000 7f1150a49700 1 -- 172.21.15.77:0/1160069505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1144003a20 con 0x7f114c164a90 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.340+0000 7f1150a49700 1 -- 172.21.15.77:0/1160069505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144003dd0 con 0x7f114c164a90 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.341+0000 7f1153caf700 1 -- 172.21.15.77:0/1160069505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c164a90 msgr2=0x7f114c06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.341+0000 7f1153caf700 1 --2- 172.21.15.77:0/1160069505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c164a90 0x7f114c06eee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1144009e50 tx=0x7f114400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:11.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.345+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faedc056180 con 0x7faee80b3b50 2024-01-15T14:35:11.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.343+0000 7f1153caf700 1 -- 172.21.15.77:0/1160069505 shutdown_connections 2024-01-15T14:35:11.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.343+0000 7f1153caf700 1 --2- 172.21.15.77:0/1160069505 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c06d4d0 0x7f114c16bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.343+0000 7f1153caf700 1 --2- 172.21.15.77:0/1160069505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c164a90 0x7f114c06eee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.343+0000 7f1153caf700 1 -- 172.21.15.77:0/1160069505 >> 172.21.15.77:0/1160069505 conn(0x7f114c0cfa30 msgr2=0x7f114c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.345+0000 7f1153caf700 1 -- 172.21.15.77:0/1160069505 shutdown_connections 2024-01-15T14:35:11.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.346+0000 7f1153caf700 1 -- 172.21.15.77:0/1160069505 wait complete. 2024-01-15T14:35:11.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.347+0000 7f1153caf700 1 Processor -- start 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.348+0000 7f1153caf700 1 -- start start 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.349+0000 7f1153caf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 0x7f114c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.349+0000 7f1153caf700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114c0b3b50 0x7f114c152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.349+0000 7f1153caf700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c164a90 0x7f114c157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.349+0000 7f1153caf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0cc440 con 0x7f114c06d4d0 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.349+0000 7f1153caf700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0cc970 con 0x7f114c0b3b50 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.349+0000 7f1153caf700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c1572a0 con 0x7f114c164a90 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 0x7f114c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 0x7f114c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57492/0 (socket says 172.21.15.77:57492) 2024-01-15T14:35:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 -- 172.21.15.77:0/582587676 learned_addr learned my addr 172.21.15.77:0/582587676 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f115224c700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c164a90 0x7f114c157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f115124a700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114c0b3b50 0x7f114c152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 -- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c164a90 msgr2=0x7f114c157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c164a90 0x7f114c157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 -- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114c0b3b50 msgr2=0x7f114c152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114c0b3b50 0x7f114c152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 -- 172.21.15.77:0/582587676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144009b30 con 0x7f114c06d4d0 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f115124a700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114c0b3b50 0x7f114c152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.350+0000 7f1151a4b700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 0x7f114c0ccf30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f114800b580 tx=0x7f114800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.351+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148011830 con 0x7f114c06d4d0 2024-01-15T14:35:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.351+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11480119b0 con 0x7f114c06d4d0 2024-01-15T14:35:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.351+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148011d60 con 0x7f114c06d4d0 2024-01-15T14:35:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.351+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c15ac60 con 0x7f114c06d4d0 2024-01-15T14:35:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.351+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c156f30 con 0x7f114c06d4d0 2024-01-15T14:35:11.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.354+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f114801a400 con 0x7f114c06d4d0 2024-01-15T14:35:11.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.355+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c157580 con 0x7f114c06d4d0 2024-01-15T14:35:11.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.355+0000 7f11427fc700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f113006b420 0x7f113006d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.355+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1148089c70 con 0x7f114c06d4d0 2024-01-15T14:35:11.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.355+0000 7f115124a700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f113006b420 0x7f113006d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.357+0000 7f115124a700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f113006b420 0x7f113006d8e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f114c164f20 tx=0x7f1144009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.365+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1148015080 con 0x7f114c06d4d0 2024-01-15T14:35:11.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.385+0000 7fabacb10700 1 Processor -- start 2024-01-15T14:35:11.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.385+0000 7fabacb10700 1 -- start start 2024-01-15T14:35:11.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7fabacb10700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faba806efc0 0x7faba806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7fabacb10700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7faba8072150 0x7faba80b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7fabacb10700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8070c60 con 0x7faba80b3b50 2024-01-15T14:35:11.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7fabacb10700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba806f3a0 con 0x7faba806efc0 2024-01-15T14:35:11.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7fabacb10700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba80b4510 con 0x7faba8072150 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7faba659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faba806efc0 0x7faba806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7faba659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faba806efc0 0x7faba806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57496/0 (socket says 172.21.15.77:57496) 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7faba659c700 1 -- 172.21.15.77:0/1652974476 learned_addr learned my addr 172.21.15.77:0/1652974476 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.387+0000 7faba6d9d700 1 --2- 172.21.15.77:0/1652974476 >> v2:172.21.15.159:3300/0 conn(0x7faba8072150 0x7faba80b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7faba659c700 1 -- 172.21.15.77:0/1652974476 >> v2:172.21.15.159:3300/0 conn(0x7faba8072150 msgr2=0x7faba80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7faba659c700 1 --2- 172.21.15.77:0/1652974476 >> v2:172.21.15.159:3300/0 conn(0x7faba8072150 0x7faba80b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7faba659c700 1 -- 172.21.15.77:0/1652974476 >> v1:172.21.15.159:6789/0 conn(0x7faba80b3b50 legacy=0x7faba816e7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7faba659c700 1 -- 172.21.15.77:0/1652974476 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba808b290 con 0x7faba806efc0 2024-01-15T14:35:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7fc56696e700 1 Processor -- start 2024-01-15T14:35:11.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7faba659c700 1 --2- 172.21.15.77:0/1652974476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba806d4d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faba80b2ad0 tx=0x7fab98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52638ac140c87b8b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.388+0000 7fc56696e700 1 -- start start 2024-01-15T14:35:11.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7faba559a700 1 -- 172.21.15.77:0/1652974476 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98004230 con 0x7faba806efc0 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7faba559a700 1 -- 172.21.15.77:0/1652974476 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98004390 con 0x7faba806efc0 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7faba559a700 1 -- 172.21.15.77:0/1652974476 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab980036a0 con 0x7faba806efc0 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 -- 172.21.15.77:0/1652974476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 msgr2=0x7faba806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 --2- 172.21.15.77:0/1652974476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba806d4d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faba80b2ad0 tx=0x7fab98009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7fc56696e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc5601446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7fc56696e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5600af950 con 0x7fc5600b5ae0 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7fc56696e700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5600afc30 con 0x7fc56006c710 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.389+0000 7fc56696e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc560161b60 con 0x7fc5600adf00 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fc564f0b700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc56006c710 0x7fc560073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:35720/0 (socket says 172.21.15.77:35720) 2024-01-15T14:35:11.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fc564f0b700 1 -- 172.21.15.77:0/3177358214 learned_addr learned my addr 172.21.15.77:0/3177358214 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fc55ffff700 1 --2- 172.21.15.77:0/3177358214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc5601446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 -- 172.21.15.77:0/1652974476 shutdown_connections 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 --2- 172.21.15.77:0/1652974476 >> v2:172.21.15.159:3300/0 conn(0x7faba8072150 0x7faba80b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 --2- 172.21.15.77:0/1652974476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba806d4d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 -- 172.21.15.77:0/1652974476 >> 172.21.15.77:0/1652974476 conn(0x7faba80cfa30 msgr2=0x7faba80cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fc55ffff700 1 -- 172.21.15.77:0/3177358214 >> v1:172.21.15.159:6789/0 conn(0x7fc56006c710 legacy=0x7fc560073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fc55ffff700 1 -- 172.21.15.77:0/3177358214 >> v1:172.21.15.77:6789/0 conn(0x7fc5600b5ae0 legacy=0x7fc56006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fc55ffff700 1 -- 172.21.15.77:0/3177358214 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600aff10 con 0x7fc5600adf00 2024-01-15T14:35:11.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 -- 172.21.15.77:0/1652974476 shutdown_connections 2024-01-15T14:35:11.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.390+0000 7fabacb10700 1 -- 172.21.15.77:0/1652974476 wait complete. 2024-01-15T14:35:11.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.392+0000 7fabacb10700 1 Processor -- start 2024-01-15T14:35:11.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.392+0000 7fabacb10700 1 -- start start 2024-01-15T14:35:11.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.392+0000 7f23a8738700 1 Processor -- start 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.392+0000 7f23a8738700 1 -- start start 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a8738700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f23a0070fd0 0x7f23a0165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a8738700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23a01663d0 0x7f23a00b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a8738700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a0072110 con 0x7f23a00cf440 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a8738700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a0166840 con 0x7f23a01663d0 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a8738700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a0071500 con 0x7f23a0070fd0 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a6cd5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23a01663d0 0x7f23a00b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a6cd5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23a01663d0 0x7f23a00b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57502/0 (socket says 172.21.15.77:57502) 2024-01-15T14:35:11.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7f23a6cd5700 1 -- 172.21.15.77:0/59875529 learned_addr learned my addr 172.21.15.77:0/59875529 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7fabacb10700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba817bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7fabacb10700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faba8072150 0x7faba817c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.393+0000 7fabacb10700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faba80b3b50 0x7faba80adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fabacb10700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba80adb50 con 0x7faba806efc0 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fabacb10700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba808a8d0 con 0x7faba8072150 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fabacb10700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba8161d10 con 0x7faba80b3b50 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba817bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba817bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57516/0 (socket says 172.21.15.77:57516) 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 -- 172.21.15.77:0/443428913 learned_addr learned my addr 172.21.15.77:0/443428913 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7f23a6cd5700 1 -- 172.21.15.77:0/59875529 >> v2:172.21.15.159:3300/0 conn(0x7f23a0070fd0 msgr2=0x7f23a0165e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7f23a6cd5700 1 --2- 172.21.15.77:0/59875529 >> v2:172.21.15.159:3300/0 conn(0x7f23a0070fd0 0x7f23a0165e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7f23a6cd5700 1 -- 172.21.15.77:0/59875529 >> v1:172.21.15.159:6789/0 conn(0x7f23a00cf440 legacy=0x7f23a006e8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7f23a6cd5700 1 -- 172.21.15.77:0/59875529 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00717e0 con 0x7f23a01663d0 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 -- 172.21.15.77:0/443428913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faba80b3b50 msgr2=0x7faba80adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faba80b3b50 0x7faba80adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 -- 172.21.15.77:0/443428913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faba8072150 msgr2=0x7faba817c020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faba8072150 0x7faba817c020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 -- 172.21.15.77:0/443428913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98009550 con 0x7faba806efc0 2024-01-15T14:35:11.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7f23a6cd5700 1 --2- 172.21.15.77:0/59875529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a00b2a40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f23a0072770 tx=0x7f239800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f404ce8cff3d4d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fc55ffff700 1 --2- 172.21.15.77:0/3177358214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc5601446a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5540097f0 tx=0x7fc554009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad950b817202c70 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7faba659c700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba817bae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab98000c00 tx=0x7fab98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fc55effd700 1 -- 172.21.15.77:0/3177358214 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554004230 con 0x7fc5600adf00 2024-01-15T14:35:11.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fc55effd700 1 -- 172.21.15.77:0/3177358214 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc554004390 con 0x7fc5600adf00 2024-01-15T14:35:11.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.394+0000 7fc55effd700 1 -- 172.21.15.77:0/3177358214 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5540036a0 con 0x7fc5600adf00 2024-01-15T14:35:11.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.395+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98003e90 con 0x7faba806efc0 2024-01-15T14:35:11.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.395+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98011630 con 0x7faba806efc0 2024-01-15T14:35:11.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.395+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab980119a0 con 0x7faba806efc0 2024-01-15T14:35:11.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.395+0000 7fc56696e700 1 -- 172.21.15.77:0/3177358214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 msgr2=0x7fc5601446a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.395+0000 7fc56696e700 1 --2- 172.21.15.77:0/3177358214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc5601446a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5540097f0 tx=0x7fc554009b00 comp rx=0 tx=0).stop 2024-01-15T14:35:11.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.396+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba8161f60 con 0x7faba806efc0 2024-01-15T14:35:11.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.396+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba80ad770 con 0x7faba806efc0 2024-01-15T14:35:11.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.397+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba80751e0 con 0x7faba806efc0 2024-01-15T14:35:11.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.398+0000 7fc56696e700 1 -- 172.21.15.77:0/3177358214 shutdown_connections 2024-01-15T14:35:11.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.398+0000 7fc56696e700 1 --2- 172.21.15.77:0/3177358214 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc5601446a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.398+0000 7fc56696e700 1 -- 172.21.15.77:0/3177358214 >> 172.21.15.77:0/3177358214 conn(0x7fc560071b90 msgr2=0x7fc560071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.399+0000 7fc56696e700 1 -- 172.21.15.77:0/3177358214 shutdown_connections 2024-01-15T14:35:11.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.399+0000 7fc56696e700 1 -- 172.21.15.77:0/3177358214 wait complete. 2024-01-15T14:35:11.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.399+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fab98021070 con 0x7faba806efc0 2024-01-15T14:35:11.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.399+0000 7f23a54d2700 1 -- 172.21.15.77:0/59875529 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239800e070 con 0x7f23a01663d0 2024-01-15T14:35:11.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7fab96ffd700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fab9006b500 0x7fab9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab9808b4c0 con 0x7faba806efc0 2024-01-15T14:35:11.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7fc56696e700 1 Processor -- start 2024-01-15T14:35:11.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7f23a54d2700 1 -- 172.21.15.77:0/59875529 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2398003a20 con 0x7f23a01663d0 2024-01-15T14:35:11.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7f23a54d2700 1 -- 172.21.15.77:0/59875529 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398003dd0 con 0x7f23a01663d0 2024-01-15T14:35:11.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7f23a8738700 1 -- 172.21.15.77:0/59875529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 msgr2=0x7f23a00b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.400+0000 7f23a8738700 1 --2- 172.21.15.77:0/59875529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a00b2a40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f23a0072770 tx=0x7f239800b330 comp rx=0 tx=0).stop 2024-01-15T14:35:11.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.401+0000 7fc56696e700 1 -- start start 2024-01-15T14:35:11.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7f23a8738700 1 -- 172.21.15.77:0/59875529 shutdown_connections 2024-01-15T14:35:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7f23a8738700 1 --2- 172.21.15.77:0/59875529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a00b2a40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7f23a8738700 1 --2- 172.21.15.77:0/59875529 >> v2:172.21.15.159:3300/0 conn(0x7f23a0070fd0 0x7f23a0165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7f23a8738700 1 -- 172.21.15.77:0/59875529 >> 172.21.15.77:0/59875529 conn(0x7f23a00b1740 msgr2=0x7f23a00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7faba5d9b700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fab9006b500 0x7fab9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc56696e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc56006c710 0x7fc560160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc56696e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc560159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc56696e700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5600b5ae0 0x7fc5601595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc56696e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600cd9e0 con 0x7fc56006c710 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc56696e700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600cdcc0 con 0x7fc5600b5ae0 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc56696e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600cdfa0 con 0x7fc5600adf00 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc564f0b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5600b5ae0 0x7fc5601595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc564f0b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5600b5ae0 0x7fc5601595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57928/0 (socket says 172.21.15.77:57928) 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc564f0b700 1 -- 172.21.15.77:0/1041257701 learned_addr learned my addr 172.21.15.77:0/1041257701 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.402+0000 7fc55f7fe700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc560159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7faba5d9b700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fab9006b500 0x7fab9006d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab9c006fd0 tx=0x7fab9c006bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7f23a8738700 1 -- 172.21.15.77:0/59875529 shutdown_connections 2024-01-15T14:35:11.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7f23a8738700 1 -- 172.21.15.77:0/59875529 wait complete. 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55ffff700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc56006c710 0x7fc560160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55f7fe700 1 -- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5600b5ae0 msgr2=0x7fc5601595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55f7fe700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5600b5ae0 0x7fc5601595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55f7fe700 1 -- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc56006c710 msgr2=0x7fc560160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55f7fe700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc56006c710 0x7fc560160b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55f7fe700 1 -- 172.21.15.77:0/1041257701 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540094a0 con 0x7fc5600adf00 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55f7fe700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc560159080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc55800cb10 tx=0x7fc55800ce20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558003c00 con 0x7fc5600adf00 2024-01-15T14:35:11.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558003d80 con 0x7fc5600adf00 2024-01-15T14:35:11.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.403+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800aae0 con 0x7fc5600adf00 2024-01-15T14:35:11.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.404+0000 7fd7d06b3700 1 Processor -- start 2024-01-15T14:35:11.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.404+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5600ce280 con 0x7fc5600adf00 2024-01-15T14:35:11.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.404+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc560161090 con 0x7fc5600adf00 2024-01-15T14:35:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.405+0000 7fd7d06b3700 1 -- start start 2024-01-15T14:35:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.405+0000 7f23a8738700 1 Processor -- start 2024-01-15T14:35:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.405+0000 7f23a8738700 1 -- start start 2024-01-15T14:35:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.406+0000 7f23a8738700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23a0070fd0 0x7f23a0145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.406+0000 7f23a8738700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23a00cf440 0x7f23a0161510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a8738700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a0161a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a8738700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0165800 con 0x7f23a01663d0 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a8738700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0146130 con 0x7f23a00cf440 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a8738700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0164bf0 con 0x7f23a0070fd0 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a5cd3700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23a00cf440 0x7f23a0161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a6cd5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a0161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a6cd5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a0161a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57526/0 (socket says 172.21.15.77:57526) 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a6cd5700 1 -- 172.21.15.77:0/1248107096 learned_addr learned my addr 172.21.15.77:0/1248107096 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7f23a64d4700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23a0070fd0 0x7f23a0145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fd7d06b3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd7c80cf440 0x7fd7c806e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fd7d06b3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c8072110 con 0x7fd7c8070fd0 2024-01-15T14:35:11.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fd7d06b3700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c806ee00 con 0x7fd7c81665a0 2024-01-15T14:35:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fd7d06b3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c8071500 con 0x7fd7c80cf440 2024-01-15T14:35:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fd7cdc4e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd7c8070fd0 0x7fd7c8165e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38654/0 (socket says 172.21.15.77:38654) 2024-01-15T14:35:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fd7cdc4e700 1 -- 172.21.15.77:0/2903099563 learned_addr learned my addr 172.21.15.77:0/2903099563 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc55800ac40 con 0x7fc5600adf00 2024-01-15T14:35:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.407+0000 7fc55cff9700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc54c06b350 0x7fc54c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc558089aa0 con 0x7fc5600adf00 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab98056320 con 0x7faba806efc0 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f3b91ab6700 1 Processor -- start 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f3b91ab6700 1 -- start start 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b91ab6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b8c0b3b50 0x7f3b8c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b91ab6700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 0x7f3b8c06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b91ab6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c06ced0 con 0x7f3b8c163a80 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b91ab6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c06da70 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b91ab6700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0b4470 con 0x7f3b8c06d4d0 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b83fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 0x7f3b8c06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b83fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 0x7f3b8c06eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53824/0 (socket says 172.21.15.77:53824) 2024-01-15T14:35:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7f3b83fff700 1 -- 172.21.15.77:0/1704343230 learned_addr learned my addr 172.21.15.77:0/1704343230 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f23a5cd3700 1 -- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23a0070fd0 msgr2=0x7f23a0145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f23a5cd3700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23a0070fd0 0x7f23a0145bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f23a5cd3700 1 -- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 msgr2=0x7f23a0161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f23a5cd3700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a0161a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f23a5cd3700 1 -- 172.21.15.77:0/1248107096 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398009b30 con 0x7f23a00cf440 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7f23a5cd3700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23a00cf440 0x7f23a0161510 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f23a0071430 tx=0x7f239c00b750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c011830 con 0x7f23a00cf440 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c011990 con 0x7f23a00cf440 2024-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c011d20 con 0x7f23a00cf440 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.406+0000 7f9b902ba700 1 Processor -- start 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.406+0000 7f9b902ba700 1 -- start start 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f9b902ba700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9b88163480 0x7f9b880b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f9b902ba700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9b880b2550 0x7f9b880b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f9b902ba700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b880b2e80 con 0x7f9b88163860 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f9b902ba700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b880b2980 con 0x7f9b88163480 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7f9b902ba700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b88164510 con 0x7f9b880b2550 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7fc55ffff700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc54c06b350 0x7fc54c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5600753f0 con 0x7fc5600adf00 2024-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fc55ffff700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc54c06b350 0x7fc54c06d810 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5540090b0 tx=0x7fc554008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9b880b2550 0x7f9b880b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9b880b2550 0x7f9b880b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57950/0 (socket says 172.21.15.77:57950) 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 -- 172.21.15.77:0/860849897 learned_addr learned my addr 172.21.15.77:0/860849897 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e056700 1 --2- 172.21.15.77:0/860849897 >> v2:172.21.15.77:3300/0 conn(0x7f9b88163480 0x7f9b880b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 -- 172.21.15.77:0/860849897 >> v2:172.21.15.77:3300/0 conn(0x7f9b88163480 msgr2=0x7f9b880b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 --2- 172.21.15.77:0/860849897 >> v2:172.21.15.77:3300/0 conn(0x7f9b88163480 0x7f9b880b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 -- 172.21.15.77:0/860849897 >> v1:172.21.15.77:6789/0 conn(0x7f9b88163860 legacy=0x7f9b880b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 -- 172.21.15.77:0/860849897 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b88160610 con 0x7f9b880b2550 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8e857700 1 --2- 172.21.15.77:0/860849897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b880b2550 0x7f9b880b06e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b880725d0 tx=0x7f9b84009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=627fc631db5c4c16 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8d054700 1 -- 172.21.15.77:0/860849897 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84004230 con 0x7f9b880b2550 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8d054700 1 -- 172.21.15.77:0/860849897 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84004390 con 0x7f9b880b2550 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7f9b8d054700 1 -- 172.21.15.77:0/860849897 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840036a0 con 0x7f9b880b2550 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.412+0000 7f9b902ba700 1 -- 172.21.15.77:0/860849897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b880b2550 msgr2=0x7f9b880b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.412+0000 7f9b902ba700 1 --2- 172.21.15.77:0/860849897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b880b2550 0x7f9b880b06e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b880725d0 tx=0x7f9b84009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7f9b902ba700 1 -- 172.21.15.77:0/860849897 shutdown_connections 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7f9b902ba700 1 --2- 172.21.15.77:0/860849897 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b880b2550 0x7f9b880b06e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7f9b902ba700 1 --2- 172.21.15.77:0/860849897 >> v2:172.21.15.77:3300/0 conn(0x7f9b88163480 0x7f9b880b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7f9b902ba700 1 -- 172.21.15.77:0/860849897 >> 172.21.15.77:0/860849897 conn(0x7f9b880b57b0 msgr2=0x7f9b880b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7f9b902ba700 1 -- 172.21.15.77:0/860849897 shutdown_connections 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7f9b902ba700 1 -- 172.21.15.77:0/860849897 wait complete. 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f3b83fff700 1 -- 172.21.15.77:0/1704343230 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c0b3b50 msgr2=0x7f3b8c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f3b83fff700 1 --2- 172.21.15.77:0/1704343230 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c0b3b50 0x7f3b8c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f3b83fff700 1 -- 172.21.15.77:0/1704343230 >> v1:172.21.15.77:6789/0 conn(0x7f3b8c163a80 legacy=0x7f3b8c163f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f3b83fff700 1 -- 172.21.15.77:0/1704343230 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c1634e0 con 0x7f3b8c06d4d0 2024-01-15T14:35:11.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f3b83fff700 1 --2- 172.21.15.77:0/1704343230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 0x7f3b8c06eee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3b740098f0 tx=0x7f3b74009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88b7a029fc0c21ec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870275485 0 0) 0x7fd7c8072110 con 0x7fd7c8070fd0 2024-01-15T14:35:11.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.408+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b40035b0 con 0x7fd7c8070fd0 2024-01-15T14:35:11.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278724618 0 0) 0x7fd7b40035b0 con 0x7fd7c8070fd0 2024-01-15T14:35:11.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 >> v2:172.21.15.77:3300/0 conn(0x7fd7c80cf440 msgr2=0x7fd7c806e8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T14:35:11.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 --2- 172.21.15.77:0/2903099563 >> v2:172.21.15.77:3300/0 conn(0x7fd7c80cf440 0x7fd7c806e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 >> v1:172.21.15.159:6789/0 conn(0x7fd7c81665a0 legacy=0x7fd7c80b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c80717e0 con 0x7fd7c8070fd0 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fd7b8001670 con 0x7fd7c8070fd0 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 >> v1:172.21.15.77:6789/0 conn(0x7fd7c8070fd0 legacy=0x7fd7c8165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7b4003a00 0x7fd7b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4004850 0x7fd7b4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8071500 con 0x7fd7b4004850 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b400c230 con 0x7fd7b4005330 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b400c3f0 con 0x7fd7b4003a00 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cec50700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cec50700 1 -- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7b4003a00 msgr2=0x7fd7b4003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cec50700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7b4003a00 0x7fd7b4003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cec50700 1 -- 172.21.15.77:0/2903099563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4004850 msgr2=0x7fd7b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cec50700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4004850 0x7fd7b4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.409+0000 7fd7cec50700 1 -- 172.21.15.77:0/2903099563 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c80717e0 con 0x7fd7b4005330 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7fd7cec50700 1 -- 172.21.15.77:0/2903099563 reap_dead start 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7fd7cec50700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7b400ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8070b70 tx=0x7fd7b4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba61a0f3d5a3777 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40052d0 con 0x7fd7b4005330 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c4005430 con 0x7fd7b4005330 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.410+0000 7fd7cd44d700 1 -- 172.21.15.77:0/2903099563 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40038a0 con 0x7fd7b4005330 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 msgr2=0x7fd7b400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7b400ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8070b70 tx=0x7fd7b4005880 comp rx=0 tx=0).stop 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2903099563 shutdown_connections 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2903099563 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7b400ba10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2903099563 >> 172.21.15.77:0/2903099563 conn(0x7fd7c80b1740 msgr2=0x7fd7c80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2903099563 shutdown_connections 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.411+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2903099563 wait complete. 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7fd7d06b3700 1 Processor -- start 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.413+0000 7fd7d06b3700 1 -- start start 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7fd7d06b3700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7c813e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7fd7d06b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 0x7fd7c808a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7fd7d06b3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7c808a600 0x7fd7c815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7fd7d06b3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c808aab0 con 0x7fd7c813e940 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7fd7d06b3700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c813ffd0 con 0x7fd7b4005330 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7fd7d06b3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c808af00 con 0x7fd7c808a600 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7fd7cdc4e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 0x7fd7c808a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7fd7cdc4e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 0x7fd7c808a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57566/0 (socket says 172.21.15.77:57566) 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7fd7cdc4e700 1 -- 172.21.15.77:0/2293744775 learned_addr learned my addr 172.21.15.77:0/2293744775 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7fd7cec50700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7c808a600 0x7fd7c815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f9b902ba700 1 Processor -- start 2024-01-15T14:35:11.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7f9b902ba700 1 -- start start 2024-01-15T14:35:11.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc558054900 con 0x7fc5600adf00 2024-01-15T14:35:11.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7f3b8affd700 1 -- 172.21.15.77:0/1704343230 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74004230 con 0x7f3b8c06d4d0 2024-01-15T14:35:11.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7f3b8affd700 1 -- 172.21.15.77:0/1704343230 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b74004390 con 0x7f3b8c06d4d0 2024-01-15T14:35:11.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7f3b8affd700 1 -- 172.21.15.77:0/1704343230 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b740036a0 con 0x7f3b8c06d4d0 2024-01-15T14:35:11.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 -- 172.21.15.77:0/1704343230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 msgr2=0x7f3b8c06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/1704343230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 0x7f3b8c06eee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3b740098f0 tx=0x7f3b74009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.414+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a0164ed0 con 0x7f23a00cf440 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.415+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a0165490 con 0x7f23a00cf440 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7fd7cdc4e700 1 -- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7c808a600 msgr2=0x7fd7c815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7fd7cdc4e700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7c808a600 0x7fd7c815f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7fd7cdc4e700 1 -- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 msgr2=0x7fd7c813e400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7fd7cdc4e700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7c813e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7fd7cdc4e700 1 -- 172.21.15.77:0/2293744775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c813ef30 con 0x7fd7c813e940 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 -- 172.21.15.77:0/1704343230 shutdown_connections 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/1704343230 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c0b3b50 0x7f3b8c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/1704343230 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c06d4d0 0x7f3b8c06eee0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 -- 172.21.15.77:0/1704343230 >> 172.21.15.77:0/1704343230 conn(0x7f3b8c0cfa30 msgr2=0x7f3b8c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 -- 172.21.15.77:0/1704343230 shutdown_connections 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b902ba700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b880b2550 0x7f9b88150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b902ba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b88163480 0x7f9b8814d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b902ba700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 0x7f9b8806e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b902ba700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b880b54f0 con 0x7f9b88163480 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b902ba700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b881508a0 con 0x7f9b88163860 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b902ba700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8814da40 con 0x7f9b880b2550 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 0x7f9b8806e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 0x7f9b8806e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57974/0 (socket says 172.21.15.77:57974) 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 -- 172.21.15.77:0/441891270 learned_addr learned my addr 172.21.15.77:0/441891270 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 -- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b880b2550 msgr2=0x7f9b88150360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b880b2550 0x7f9b88150360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 -- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b88163480 msgr2=0x7f9b8814d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b88163480 0x7f9b8814d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 -- 172.21.15.77:0/441891270 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84009550 con 0x7f9b88163860 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.416+0000 7f9b8e857700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 0x7f9b8806e700 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b84004260 tx=0x7f9b84004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84003d00 con 0x7f9b88163860 2024-01-15T14:35:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8401f070 con 0x7f9b88163860 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84011870 con 0x7f9b88163860 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b88150ff0 con 0x7f9b88163860 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b881515b0 con 0x7f9b88163860 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7fd7cdc4e700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 0x7fd7c808a0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c680 tx=0x7fd7b800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.418+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8004c80 con 0x7fd7c813e940 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.418+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b8004de0 con 0x7fd7c813e940 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.418+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8007310 con 0x7fd7c813e940 2024-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.419+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a00751e0 con 0x7f23a00cf440 2024-01-15T14:35:11.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.417+0000 7f3b91ab6700 1 -- 172.21.15.77:0/1704343230 wait complete. 2024-01-15T14:35:11.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.419+0000 7f3b91ab6700 1 Processor -- start 2024-01-15T14:35:11.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.419+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c813f210 con 0x7fd7c813e940 2024-01-15T14:35:11.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.419+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c813f7d0 con 0x7fd7c813e940 2024-01-15T14:35:11.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.419+0000 7f3b91ab6700 1 -- start start 2024-01-15T14:35:11.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c80751e0 con 0x7fd7c813e940 2024-01-15T14:35:11.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7f3b91ab6700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b8c06d4d0 0x7f3b8c157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7f3b91ab6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c0b3b50 0x7f3b8c159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7f3b91ab6700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c163a80 0x7f3b8c1f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7f3b91ab6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c201f40 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7f3b91ab6700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c158390 con 0x7f3b8c06d4d0 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.420+0000 7f3b91ab6700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c2018f0 con 0x7f3b8c163a80 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f3b8bfff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c163a80 0x7f3b8c1f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f3b8b7fe700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b8c06d4d0 0x7f3b8c157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f3b8bfff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c163a80 0x7f3b8c1f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53838/0 (socket says 172.21.15.77:53838) 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f3b8bfff700 1 -- 172.21.15.77:0/4131464291 learned_addr learned my addr 172.21.15.77:0/4131464291 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd7b8007470 con 0x7fd7c813e940 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7fd7beffd700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd7ac06b420 0x7fd7ac06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7b80898e0 con 0x7fd7c813e940 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9b84021030 con 0x7f9b88163860 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f9b7effd700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b6c06b3a0 0x7f9b6c06d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b8408b0a0 con 0x7f9b88163860 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.421+0000 7f9b8e056700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b6c06b3a0 0x7f9b6c06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.422+0000 7f3b83fff700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c0b3b50 0x7f3b8c159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.422+0000 7f3b83fff700 1 -- 172.21.15.77:0/4131464291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c163a80 msgr2=0x7f3b8c1f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.422+0000 7f3b83fff700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c163a80 0x7f3b8c1f6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.422+0000 7f3b83fff700 1 -- 172.21.15.77:0/4131464291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b8c06d4d0 msgr2=0x7f3b8c157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.422+0000 7f3b83fff700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b8c06d4d0 0x7f3b8c157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.422+0000 7f3b83fff700 1 -- 172.21.15.77:0/4131464291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74009550 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.423+0000 7f3b83fff700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c0b3b50 0x7f3b8c159d10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3b74015000 tx=0x7f3b74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.427+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74003e60 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.427+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7400b660 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.427+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7400ba10 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.424+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f239c01a400 con 0x7f23a00cf440 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.424+0000 7f2396ffd700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f238406b350 0x7f238406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.424+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f239c089a80 con 0x7f23a00cf440 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.425+0000 7f23a64d4700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f238406b350 0x7f238406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.425+0000 7f23a64d4700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f238406b350 0x7f238406d810 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23a006ece0 tx=0x7f2390009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.427+0000 7fd7ce44f700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd7ac06b420 0x7fd7ac06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.428+0000 7fd7ce44f700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd7ac06b420 0x7fd7ac06d8e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd7c80af500 tx=0x7fd7c0009230 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.423+0000 7f9b8e056700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b6c06b3a0 0x7f9b6c06d860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009770 tx=0x7f9b78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.424+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b88067f60 con 0x7f9b88163860 2024-01-15T14:35:11.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.428+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f239c057f00 con 0x7f23a00cf440 2024-01-15T14:35:11.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.428+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7b8054730 con 0x7fd7c813e940 2024-01-15T14:35:11.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.427+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c201bd0 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.427+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c158670 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.431+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c0751e0 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.431+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3b74021070 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.431+0000 7f3b88ff9700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b7806b530 0x7f3b7806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:11.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.432+0000 7f3b8b7fe700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b7806b530 0x7f3b7806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:11.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.433+0000 7f3b8b7fe700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b7806b530 0x7f3b7806d9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009a10 tx=0x7f3b7c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:11.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.433+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b740193c0 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.441+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b84055fa0 con 0x7f9b88163860 2024-01-15T14:35:11.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.441+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b740560e0 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.744+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7faec80055a0 con 0x7faee80b3b50 2024-01-15T14:35:11.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.744+0000 7faecf7fe700 1 -- 172.21.15.77:0/39864631 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faedc0597a0 con 0x7faee80b3b50 2024-01-15T14:35:11.747 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.748+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7faed406b420 msgr2=0x7faed406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.748+0000 7faeed7ff700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7faed406b420 0x7faed406d8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faed8006b20 tx=0x7faed8008040 comp rx=0 tx=0).stop 2024-01-15T14:35:11.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.748+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faee80b3b50 msgr2=0x7faee8136af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.748+0000 7faeed7ff700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faee80b3b50 0x7faee8136af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faedc016000 tx=0x7faedc004750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 shutdown_connections 2024-01-15T14:35:11.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7faed406b420 0x7faed406d8e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faee80714d0 0x7faee814d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faee80d0520 0x7faee8152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 --2- 172.21.15.77:0/39864631 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faee80b3b50 0x7faee8136af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 >> 172.21.15.77:0/39864631 conn(0x7faee80cfa30 msgr2=0x7faee815f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 shutdown_connections 2024-01-15T14:35:11.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.749+0000 7faeed7ff700 1 -- 172.21.15.77:0/39864631 wait complete. 2024-01-15T14:35:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.764+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f114c06eac0 con 0x7f114c06d4d0 2024-01-15T14:35:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.765+0000 7f11427fc700 1 -- 172.21.15.77:0/582587676 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1148054b50 con 0x7f114c06d4d0 2024-01-15T14:35:11.765 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-01-15T14:35:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.768+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f113006b420 msgr2=0x7f113006d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.768+0000 7f1153caf700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f113006b420 0x7f113006d8e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f114c164f20 tx=0x7f1144009000 comp rx=0 tx=0).stop 2024-01-15T14:35:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.768+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 msgr2=0x7f114c0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.768+0000 7f1153caf700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 0x7f114c0ccf30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f114800b580 tx=0x7f114800b940 comp rx=0 tx=0).stop 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 shutdown_connections 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f113006b420 0x7f113006d8e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f114c164a90 0x7f114c157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f114c06d4d0 0x7f114c0ccf30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 --2- 172.21.15.77:0/582587676 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f114c0b3b50 0x7f114c152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 >> 172.21.15.77:0/582587676 conn(0x7f114c0cfa30 msgr2=0x7f114c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 shutdown_connections 2024-01-15T14:35:11.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.770+0000 7f1153caf700 1 -- 172.21.15.77:0/582587676 wait complete. 2024-01-15T14:35:11.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-01-15T14:35:11.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.796+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc560067f60 con 0x7fc5600adf00 2024-01-15T14:35:11.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.796+0000 7fc55cff9700 1 -- 172.21.15.77:0/1041257701 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc558057f20 con 0x7fc5600adf00 2024-01-15T14:35:11.798 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-01-15T14:35:11.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.800+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc54c06b350 msgr2=0x7fc54c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.800+0000 7fc56696e700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc54c06b350 0x7fc54c06d810 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5540090b0 tx=0x7fc554008030 comp rx=0 tx=0).stop 2024-01-15T14:35:11.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.800+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 msgr2=0x7fc560159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.800+0000 7fc56696e700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc560159080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc55800cb10 tx=0x7fc55800ce20 comp rx=0 tx=0).stop 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 shutdown_connections 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc54c06b350 0x7fc54c06d810 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc5600b5ae0 0x7fc5601595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc56006c710 0x7fc560160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 --2- 172.21.15.77:0/1041257701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5600adf00 0x7fc560159080 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 >> 172.21.15.77:0/1041257701 conn(0x7fc560071b90 msgr2=0x7fc5600b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 shutdown_connections 2024-01-15T14:35:11.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.802+0000 7fc56696e700 1 -- 172.21.15.77:0/1041257701 wait complete. 2024-01-15T14:35:11.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.803+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7faba80d0760 con 0x7faba806efc0 2024-01-15T14:35:11.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.804+0000 7fab96ffd700 1 -- 172.21.15.77:0/443428913 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab98059940 con 0x7faba806efc0 2024-01-15T14:35:11.804 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-01-15T14:35:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.808+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fab9006b500 msgr2=0x7fab9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.808+0000 7fabacb10700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fab9006b500 0x7fab9006d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab9c006fd0 tx=0x7fab9c006bc0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.808+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 msgr2=0x7faba817bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.808+0000 7fabacb10700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba817bae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab98000c00 tx=0x7fab98004750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 shutdown_connections 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fab9006b500 0x7fab9006d9c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7faba80b3b50 0x7faba80adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba806efc0 0x7faba817bae0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 --2- 172.21.15.77:0/443428913 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7faba8072150 0x7faba817c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 >> 172.21.15.77:0/443428913 conn(0x7faba80cfa30 msgr2=0x7faba816da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 shutdown_connections 2024-01-15T14:35:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.811+0000 7fabacb10700 1 -- 172.21.15.77:0/443428913 wait complete. 2024-01-15T14:35:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.812+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3b8c201a30 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-01-15T14:35:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.816+0000 7f3b88ff9700 1 -- 172.21.15.77:0/4131464291 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b8c201a30 con 0x7f3b8c0b3b50 2024-01-15T14:35:11.816 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.819+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b7806b530 msgr2=0x7f3b7806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.819+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b7806b530 0x7f3b7806d9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009a10 tx=0x7f3b7c009190 comp rx=0 tx=0).stop 2024-01-15T14:35:11.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.819+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c0b3b50 msgr2=0x7f3b8c159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.819+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c0b3b50 0x7f3b8c159d10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3b74015000 tx=0x7f3b74004750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.820+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd7c8072a80 con 0x7fd7c813e940 2024-01-15T14:35:11.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.820+0000 7fd7beffd700 1 -- 172.21.15.77:0/2293744775 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7b8001a90 con 0x7fd7c813e940 2024-01-15T14:35:11.820 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2024-01-15T14:35:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.821+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 shutdown_connections 2024-01-15T14:35:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.821+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b7806b530 0x7f3b7806d9f0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009a10 tx=0x7f3b7c009190 comp rx=0 tx=0).stop 2024-01-15T14:35:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.821+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8c163a80 0x7f3b8c1f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.821+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b8c06d4d0 0x7f3b8c157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.821+0000 7f3b91ab6700 1 --2- 172.21.15.77:0/4131464291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c0b3b50 0x7f3b8c159d10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.821+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 >> 172.21.15.77:0/4131464291 conn(0x7f3b8c0cfa30 msgr2=0x7f3b8c15bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.822+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 shutdown_connections 2024-01-15T14:35:11.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.822+0000 7f3b91ab6700 1 -- 172.21.15.77:0/4131464291 wait complete. 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd7ac06b420 msgr2=0x7fd7ac06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd7ac06b420 0x7fd7ac06d8e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd7c80af500 tx=0x7fd7c0009230 comp rx=0 tx=0).stop 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 msgr2=0x7fd7c808a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 0x7fd7c808a0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c680 tx=0x7fd7b800c990 comp rx=0 tx=0).stop 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 shutdown_connections 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd7ac06b420 0x7fd7ac06d8e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7c808a600 0x7fd7c815f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd7b4005330 0x7fd7c813e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 --2- 172.21.15.77:0/2293744775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7c813e940 0x7fd7c808a0c0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c680 tx=0x7fd7b800c990 comp rx=0 tx=0).stop 2024-01-15T14:35:11.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.823+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 >> 172.21.15.77:0/2293744775 conn(0x7fd7c80b1740 msgr2=0x7fd7c8162730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.824+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 shutdown_connections 2024-01-15T14:35:11.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.824+0000 7fd7d06b3700 1 -- 172.21.15.77:0/2293744775 wait complete. 2024-01-15T14:35:11.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-01-15T14:35:11.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.835+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f23a00b4a70 con 0x7f23a00cf440 2024-01-15T14:35:11.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-01-15T14:35:11.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.835+0000 7f2396ffd700 1 -- 172.21.15.77:0/1248107096 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f239c0580e0 con 0x7f23a00cf440 2024-01-15T14:35:11.836 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-01-15T14:35:11.839 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f238406b350 msgr2=0x7f238406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f238406b350 0x7f238406d810 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23a006ece0 tx=0x7f2390009190 comp rx=0 tx=0).stop 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23a00cf440 msgr2=0x7f23a0161510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23a00cf440 0x7f23a0161510 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f23a0071430 tx=0x7f239c00b750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 shutdown_connections 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f238406b350 0x7f238406d810 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23a0070fd0 0x7f23a0145bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a01663d0 0x7f23a0161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 --2- 172.21.15.77:0/1248107096 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23a00cf440 0x7f23a0161510 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 >> 172.21.15.77:0/1248107096 conn(0x7f23a00b1740 msgr2=0x7f23a00afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 shutdown_connections 2024-01-15T14:35:11.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.839+0000 7f23a8738700 1 -- 172.21.15.77:0/1248107096 wait complete. 2024-01-15T14:35:11.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.844+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9b880aed10 con 0x7f9b88163860 2024-01-15T14:35:11.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.845+0000 7f9b7effd700 1 -- 172.21.15.77:0/441891270 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9b84003e60 con 0x7f9b88163860 2024-01-15T14:35:11.845 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b6c06b3a0 msgr2=0x7f9b6c06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b6c06b3a0 0x7f9b6c06d860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009770 tx=0x7f9b78008040 comp rx=0 tx=0).stop 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 msgr2=0x7f9b8806e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 0x7f9b8806e700 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b84004260 tx=0x7f9b84004750 comp rx=0 tx=0).stop 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 shutdown_connections 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b6c06b3a0 0x7f9b6c06d860 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b880b2550 0x7f9b88150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b88163860 0x7f9b8806e700 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 --2- 172.21.15.77:0/441891270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b88163480 0x7f9b8814d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 >> 172.21.15.77:0/441891270 conn(0x7f9b880b57b0 msgr2=0x7f9b8806d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 shutdown_connections 2024-01-15T14:35:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:11.847+0000 7f9b902ba700 1 -- 172.21.15.77:0/441891270 wait complete. 2024-01-15T14:35:11.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-01-15T14:35:11.860 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-01-15T14:35:12.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T14:35:12.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T14:35:12.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T14:35:12.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T14:35:12.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T14:35:12.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T14:35:12.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T14:35:12.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T14:35:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.930+0000 7fef00a3d700 1 Processor -- start 2024-01-15T14:35:12.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.936+0000 7fef00a3d700 1 -- start start 2024-01-15T14:35:12.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.937+0000 7fef00a3d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feefc15e080 0x7feefc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:12.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.937+0000 7fef00a3d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feefc06c710 0x7feefc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.937+0000 7fef00a3d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feefc0ae430 con 0x7feefc0af4d0 2024-01-15T14:35:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.937+0000 7fef00a3d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feefc0af0b0 con 0x7feefc06c710 2024-01-15T14:35:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.937+0000 7fef00a3d700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feefc0b05a0 con 0x7feefc15e080 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feefc15e080 0x7feefc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7feefc15e080 0x7feefc0b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57986/0 (socket says 172.21.15.77:57986) 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 -- 172.21.15.77:0/4025124389 learned_addr learned my addr 172.21.15.77:0/4025124389 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 -- 172.21.15.77:0/4025124389 >> v2:172.21.15.77:3300/0 conn(0x7feefc06c710 msgr2=0x7feefc0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 --2- 172.21.15.77:0/4025124389 >> v2:172.21.15.77:3300/0 conn(0x7feefc06c710 0x7feefc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 -- 172.21.15.77:0/4025124389 >> v1:172.21.15.77:6789/0 conn(0x7feefc0af4d0 legacy=0x7feefc06c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.938+0000 7feefa59c700 1 -- 172.21.15.77:0/4025124389 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc0ae710 con 0x7feefc15e080 2024-01-15T14:35:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.939+0000 7feefa59c700 1 --2- 172.21.15.77:0/4025124389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc15e080 0x7feefc0b5050 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feefc072a90 tx=0x7feeec009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90337a9c86a1e632 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.940+0000 7feef959a700 1 -- 172.21.15.77:0/4025124389 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec004230 con 0x7feefc15e080 2024-01-15T14:35:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.940+0000 7feef959a700 1 -- 172.21.15.77:0/4025124389 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec004390 con 0x7feefc15e080 2024-01-15T14:35:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.940+0000 7feef959a700 1 -- 172.21.15.77:0/4025124389 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec0036a0 con 0x7feefc15e080 2024-01-15T14:35:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.941+0000 7fef00a3d700 1 -- 172.21.15.77:0/4025124389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc15e080 msgr2=0x7feefc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.941+0000 7fef00a3d700 1 --2- 172.21.15.77:0/4025124389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc15e080 0x7feefc0b5050 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feefc072a90 tx=0x7feeec009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.942+0000 7fef00a3d700 1 -- 172.21.15.77:0/4025124389 shutdown_connections 2024-01-15T14:35:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.942+0000 7fef00a3d700 1 --2- 172.21.15.77:0/4025124389 >> v2:172.21.15.77:3300/0 conn(0x7feefc06c710 0x7feefc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.942+0000 7fef00a3d700 1 --2- 172.21.15.77:0/4025124389 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc15e080 0x7feefc0b5050 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.942+0000 7fef00a3d700 1 -- 172.21.15.77:0/4025124389 >> 172.21.15.77:0/4025124389 conn(0x7feefc071b90 msgr2=0x7feefc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.942+0000 7fef00a3d700 1 -- 172.21.15.77:0/4025124389 shutdown_connections 2024-01-15T14:35:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.942+0000 7fef00a3d700 1 -- 172.21.15.77:0/4025124389 wait complete. 2024-01-15T14:35:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.944+0000 7fef00a3d700 1 Processor -- start 2024-01-15T14:35:12.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.945+0000 7fef00a3d700 1 -- start start 2024-01-15T14:35:12.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.946+0000 7fef00a3d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 0x7feefc157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:12.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.946+0000 7fef00a3d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feefc15e080 0x7feefc200cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:12.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.946+0000 7fef00a3d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc150330 0x7feefc155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:12.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.946+0000 7fef00a3d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc2011f0 con 0x7feefc06c710 2024-01-15T14:35:12.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.946+0000 7fef00a3d700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc158480 con 0x7feefc150330 2024-01-15T14:35:12.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.946+0000 7fef00a3d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc157be0 con 0x7feefc15e080 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 0x7feefc157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 0x7feefc157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57598/0 (socket says 172.21.15.77:57598) 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 -- 172.21.15.77:0/1724450789 learned_addr learned my addr 172.21.15.77:0/1724450789 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 -- 172.21.15.77:0/1724450789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feefc15e080 msgr2=0x7feefc200cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feefc15e080 0x7feefc200cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 -- 172.21.15.77:0/1724450789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc150330 msgr2=0x7feefc155c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc150330 0x7feefc155c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:12.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.947+0000 7feefa59c700 1 -- 172.21.15.77:0/1724450789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec009550 con 0x7feefc06c710 2024-01-15T14:35:12.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.948+0000 7feefa59c700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 0x7feefc157500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feeec009fd0 tx=0x7feeec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:12.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.949+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec011630 con 0x7feefc06c710 2024-01-15T14:35:12.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.949+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec011790 con 0x7feefc06c710 2024-01-15T14:35:12.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.949+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec011b00 con 0x7feefc06c710 2024-01-15T14:35:12.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.949+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feefc157ec0 con 0x7feefc06c710 2024-01-15T14:35:12.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.949+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feefc1540b0 con 0x7feefc06c710 2024-01-15T14:35:12.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.955+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feeec021070 con 0x7feefc06c710 2024-01-15T14:35:12.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.955+0000 7feeeaffd700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7feedc06b450 0x7feedc06d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:12.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.956+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feeec08b090 con 0x7feefc06c710 2024-01-15T14:35:12.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.958+0000 7feef9d9b700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7feedc06b450 0x7feedc06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:12.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.959+0000 7feef9d9b700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7feedc06b450 0x7feedc06d910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feefc155bb0 tx=0x7feef0006c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:12.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.961+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee00052f0 con 0x7feefc06c710 2024-01-15T14:35:12.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.970+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feeec055f70 con 0x7feefc06c710 2024-01-15T14:35:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.998+0000 7f8cafe66700 1 Processor -- start 2024-01-15T14:35:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:12.999+0000 7f8cafe66700 1 -- start start 2024-01-15T14:35:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.000+0000 7f8cafe66700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ca81603d0 0x7f8ca815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.000+0000 7f8cafe66700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca815fd70 0x7f8ca8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.000+0000 7f8cafe66700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca81601c0 con 0x7f8ca8145740 2024-01-15T14:35:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.000+0000 7f8cafe66700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca8070ac0 con 0x7f8ca81603d0 2024-01-15T14:35:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.000+0000 7f8cafe66700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca806d670 con 0x7f8ca815fd70 2024-01-15T14:35:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.001+0000 7f8cadc02700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8ca8145740 0x7f8ca806e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:35738/0 (socket says 172.21.15.77:35738) 2024-01-15T14:35:13.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.001+0000 7f8cadc02700 1 -- 172.21.15.77:0/3462137374 learned_addr learned my addr 172.21.15.77:0/3462137374 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:13.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.001+0000 7f8cae403700 1 --2- 172.21.15.77:0/3462137374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca815fd70 0x7f8ca8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.001+0000 7f8cad401700 1 --2- 172.21.15.77:0/3462137374 >> v2:172.21.15.77:3300/0 conn(0x7f8ca81603d0 0x7f8ca815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.002+0000 7f8cad401700 1 -- 172.21.15.77:0/3462137374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca815fd70 msgr2=0x7f8ca8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.002+0000 7f8cad401700 1 --2- 172.21.15.77:0/3462137374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca815fd70 0x7f8ca8070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.002+0000 7f8cad401700 1 -- 172.21.15.77:0/3462137374 >> v1:172.21.15.159:6789/0 conn(0x7f8ca8145740 legacy=0x7f8ca806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.002+0000 7f8cad401700 1 -- 172.21.15.77:0/3462137374 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca806d950 con 0x7f8ca81603d0 2024-01-15T14:35:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.002+0000 7f8cad401700 1 --2- 172.21.15.77:0/3462137374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca815f830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009840 tx=0x7f8ca4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578146dd72a334e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.003+0000 7f8cacc00700 1 -- 172.21.15.77:0/3462137374 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4004230 con 0x7f8ca81603d0 2024-01-15T14:35:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.003+0000 7f8cacc00700 1 -- 172.21.15.77:0/3462137374 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca4004390 con 0x7f8ca81603d0 2024-01-15T14:35:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.003+0000 7f8cacc00700 1 -- 172.21.15.77:0/3462137374 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40036a0 con 0x7f8ca81603d0 2024-01-15T14:35:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.004+0000 7f8cafe66700 1 -- 172.21.15.77:0/3462137374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 msgr2=0x7f8ca815f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.004+0000 7f8cafe66700 1 --2- 172.21.15.77:0/3462137374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca815f830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009840 tx=0x7f8ca4009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.004+0000 7f8cafe66700 1 -- 172.21.15.77:0/3462137374 shutdown_connections 2024-01-15T14:35:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.004+0000 7f8cafe66700 1 --2- 172.21.15.77:0/3462137374 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca815fd70 0x7f8ca8070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.004+0000 7f8cafe66700 1 --2- 172.21.15.77:0/3462137374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca815f830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.004+0000 7f8cafe66700 1 -- 172.21.15.77:0/3462137374 >> 172.21.15.77:0/3462137374 conn(0x7f8ca80b1740 msgr2=0x7f8ca80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.005+0000 7f8cafe66700 1 -- 172.21.15.77:0/3462137374 shutdown_connections 2024-01-15T14:35:13.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.005+0000 7f8cafe66700 1 -- 172.21.15.77:0/3462137374 wait complete. 2024-01-15T14:35:13.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.007+0000 7f8cafe66700 1 Processor -- start 2024-01-15T14:35:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.007+0000 7f8cafe66700 1 -- start start 2024-01-15T14:35:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.008+0000 7f8cafe66700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca8145740 0x7f8ca8138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.008+0000 7f8cafe66700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ca815fd70 0x7f8ca813b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.008+0000 7f8cafe66700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca813bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.008+0000 7f8cafe66700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca81f4e30 con 0x7f8ca81603d0 2024-01-15T14:35:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.008+0000 7f8cafe66700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca813bff0 con 0x7f8ca815fd70 2024-01-15T14:35:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.008+0000 7f8cafe66700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca81f41b0 con 0x7f8ca8145740 2024-01-15T14:35:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca813bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca813bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57624/0 (socket says 172.21.15.77:57624) 2024-01-15T14:35:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 -- 172.21.15.77:0/877640140 learned_addr learned my addr 172.21.15.77:0/877640140 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 -- 172.21.15.77:0/877640140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca8145740 msgr2=0x7f8ca8138210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca8145740 0x7f8ca8138210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 -- 172.21.15.77:0/877640140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ca815fd70 msgr2=0x7f8ca813b570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.009+0000 7f8cae403700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ca815fd70 0x7f8ca813b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8cae403700 1 -- 172.21.15.77:0/877640140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40094a0 con 0x7f8ca81603d0 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8cae403700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca813bab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800e810 tx=0x7f8c9800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98013610 con 0x7f8ca81603d0 2024-01-15T14:35:13.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c98013770 con 0x7f8ca81603d0 2024-01-15T14:35:13.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98013b20 con 0x7f8ca81603d0 2024-01-15T14:35:13.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca81f4490 con 0x7f8ca81603d0 2024-01-15T14:35:13.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.010+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca81f4a50 con 0x7f8ca81603d0 2024-01-15T14:35:13.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.013+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca80cf8a0 con 0x7f8ca81603d0 2024-01-15T14:35:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.015+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8c98013c80 con 0x7f8ca81603d0 2024-01-15T14:35:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.015+0000 7f8c9e7fc700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c9406b500 0x7f8c9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.015+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c980897e0 con 0x7f8ca81603d0 2024-01-15T14:35:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.016+0000 7f8cadc02700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c9406b500 0x7f8c9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.017+0000 7f8cadc02700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c9406b500 0x7f8c9406d9c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8138c60 tx=0x7f8ca0001930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.017+0000 7f6cc72aa700 1 Processor -- start 2024-01-15T14:35:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.018+0000 7f6cc72aa700 1 -- start start 2024-01-15T14:35:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.019+0000 7f6cc72aa700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6cc015d6a0 0x7f6cc00b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.019+0000 7f6cc72aa700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc00720e0 con 0x7f6cc0071be0 2024-01-15T14:35:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.019+0000 7f6cc72aa700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc00723c0 con 0x7f6cc00b4090 2024-01-15T14:35:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.019+0000 7f6cc72aa700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc0072d60 con 0x7f6cc015d6a0 2024-01-15T14:35:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.020+0000 7f6cc5847700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6cc0071be0 0x7f6cc0140fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38670/0 (socket says 172.21.15.77:38670) 2024-01-15T14:35:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.020+0000 7f6cc5847700 1 -- 172.21.15.77:0/3230909657 learned_addr learned my addr 172.21.15.77:0/3230909657 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.020+0000 7f6cc5046700 1 --2- 172.21.15.77:0/3230909657 >> v2:172.21.15.159:3300/0 conn(0x7f6cc015d6a0 0x7f6cc00b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.020+0000 7f6cb7fff700 1 -- 172.21.15.77:0/3230909657 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205489031 0 0) 0x7f6cc00723c0 con 0x7f6cc00b4090 2024-01-15T14:35:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.021+0000 7f6cb7fff700 1 -- 172.21.15.77:0/3230909657 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cac0057c0 con 0x7f6cc00b4090 2024-01-15T14:35:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.021+0000 7f6cc5046700 1 -- 172.21.15.77:0/3230909657 >> v1:172.21.15.159:6789/0 conn(0x7f6cc00b4090 legacy=0x7f6cc00714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.021+0000 7f6cc5046700 1 -- 172.21.15.77:0/3230909657 >> v1:172.21.15.77:6789/0 conn(0x7f6cc0071be0 legacy=0x7f6cc0140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.021+0000 7f6cc5046700 1 -- 172.21.15.77:0/3230909657 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc00726a0 con 0x7f6cc015d6a0 2024-01-15T14:35:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.022+0000 7f6cc5046700 1 --2- 172.21.15.77:0/3230909657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc015d6a0 0x7f6cc00b3b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00a9e0 tx=0x7f6cbc00acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6823bbf1e382dd05 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.023+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c98054640 con 0x7f8ca81603d0 2024-01-15T14:35:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.023+0000 7f6cb7fff700 1 -- 172.21.15.77:0/3230909657 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc00cb30 con 0x7f6cc015d6a0 2024-01-15T14:35:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.023+0000 7f6cb7fff700 1 -- 172.21.15.77:0/3230909657 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc00cc90 con 0x7f6cc015d6a0 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.023+0000 7f6cb7fff700 1 -- 172.21.15.77:0/3230909657 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc00ae20 con 0x7f6cc015d6a0 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 -- 172.21.15.77:0/3230909657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc015d6a0 msgr2=0x7f6cc00b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/3230909657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc015d6a0 0x7f6cc00b3b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00a9e0 tx=0x7f6cbc00acf0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 -- 172.21.15.77:0/3230909657 shutdown_connections 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/3230909657 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc015d6a0 0x7f6cc00b3b50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 -- 172.21.15.77:0/3230909657 >> 172.21.15.77:0/3230909657 conn(0x7f6cc00cfa30 msgr2=0x7f6cc00cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 -- 172.21.15.77:0/3230909657 shutdown_connections 2024-01-15T14:35:13.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.024+0000 7f6cc72aa700 1 -- 172.21.15.77:0/3230909657 wait complete. 2024-01-15T14:35:13.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.026+0000 7f6cc72aa700 1 Processor -- start 2024-01-15T14:35:13.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.026+0000 7f6cc72aa700 1 -- start start 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc72aa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc0071be0 0x7f6cc015c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc72aa700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc00b4090 0x7f6cc01f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc72aa700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cc015d6a0 0x7f6cc013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc72aa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc0157880 con 0x7f6cc0071be0 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc72aa700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc0157b60 con 0x7f6cc00b4090 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc72aa700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc0157e40 con 0x7f6cc015d6a0 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc4845700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc00b4090 0x7f6cc01f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc4845700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc00b4090 0x7f6cc01f80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:58010/0 (socket says 172.21.15.77:58010) 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc4845700 1 -- 172.21.15.77:0/201247842 learned_addr learned my addr 172.21.15.77:0/201247842 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc5847700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cc015d6a0 0x7f6cc013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.028+0000 7f6cc5046700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc0071be0 0x7f6cc015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.029+0000 7f6cc5847700 1 -- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc00b4090 msgr2=0x7f6cc01f80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.029+0000 7f6cc5847700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc00b4090 0x7f6cc01f80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.029+0000 7f6cc5847700 1 -- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc0071be0 msgr2=0x7f6cc015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.029+0000 7f6cc5847700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc0071be0 0x7f6cc015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.029+0000 7f6cc5847700 1 -- 172.21.15.77:0/201247842 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc00a640 con 0x7f6cc015d6a0 2024-01-15T14:35:13.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.031+0000 7f6cc5847700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cc015d6a0 0x7f6cc013cfe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8001a20 tx=0x7f6cb8001d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.032+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8006550 con 0x7f6cc015d6a0 2024-01-15T14:35:13.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.032+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb80066b0 con 0x7f6cc015d6a0 2024-01-15T14:35:13.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.032+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8006a20 con 0x7f6cc015d6a0 2024-01-15T14:35:13.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.032+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc0158120 con 0x7f6cc015d6a0 2024-01-15T14:35:13.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.032+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc013d520 con 0x7f6cc015d6a0 2024-01-15T14:35:13.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.037+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6cb8006b80 con 0x7f6cc015d6a0 2024-01-15T14:35:13.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.037+0000 7f6cb5ffb700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6ca406b350 0x7f6ca406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.037+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cb80899f0 con 0x7f6cc015d6a0 2024-01-15T14:35:13.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.038+0000 7f6cc5046700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6ca406b350 0x7f6ca406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.039+0000 7f6cc5046700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6ca406b350 0x7f6ca406d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc008010 tx=0x7f6cbc01a040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.039+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca00052f0 con 0x7f6cc015d6a0 2024-01-15T14:35:13.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.046+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6cb80582e0 con 0x7f6cc015d6a0 2024-01-15T14:35:13.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.075+0000 7f7d349b8700 1 Processor -- start 2024-01-15T14:35:13.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.075+0000 7f7d349b8700 1 -- start start 2024-01-15T14:35:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.077+0000 7f7d349b8700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7d300b3b50 0x7f7d300b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.077+0000 7f7d349b8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d30070ac0 con 0x7f7d3015e1f0 2024-01-15T14:35:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.077+0000 7f7d349b8700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d300b4470 con 0x7f7d3006d1c0 2024-01-15T14:35:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.077+0000 7f7d349b8700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d30071ef0 con 0x7f7d300b3b50 2024-01-15T14:35:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.077+0000 7f7d2dd9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d3015e1f0 0x7f7d300717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38686/0 (socket says 172.21.15.77:38686) 2024-01-15T14:35:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.077+0000 7f7d2dd9b700 1 -- 172.21.15.77:0/2256866965 learned_addr learned my addr 172.21.15.77:0/2256866965 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2e59c700 1 --2- 172.21.15.77:0/2256866965 >> v2:172.21.15.159:3300/0 conn(0x7f7d300b3b50 0x7f7d300b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029944852 0 0) 0x7f7d30070ac0 con 0x7f7d3015e1f0 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d180035b0 con 0x7f7d3015e1f0 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283374687 0 0) 0x7f7d180035b0 con 0x7f7d3015e1f0 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 >> v2:172.21.15.159:3300/0 conn(0x7f7d300b3b50 msgr2=0x7f7d300b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 --2- 172.21.15.77:0/2256866965 >> v2:172.21.15.159:3300/0 conn(0x7f7d300b3b50 0x7f7d300b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 >> v1:172.21.15.159:6789/0 conn(0x7f7d3006d1c0 legacy=0x7f7d300adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d30070d10 con 0x7f7d3015e1f0 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f7d20001670 con 0x7f7d3015e1f0 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 >> v1:172.21.15.77:6789/0 conn(0x7f7d3015e1f0 legacy=0x7f7d300717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18003a00 0x7f7d18003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d18004850 0x7f7d18004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d18005210 0x7f7d1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30071ef0 con 0x7f7d18005210 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1800c0b0 con 0x7f7d18004850 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.078+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1800c270 con 0x7f7d18003a00 2024-01-15T14:35:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d18005210 0x7f7d1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/2256866965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18003a00 msgr2=0x7f7d18003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:13.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18003a00 0x7f7d18003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/2256866965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d18004850 msgr2=0x7f7d18004cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d18004850 0x7f7d18004cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/2256866965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d30070d10 con 0x7f7d18005210 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/2256866965 reap_dead start 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.080+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d18005210 0x7f7d1800ba10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800c190 tx=0x7f7d18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214bdf3bf32a9f26 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7fe5382be700 1 Processor -- start 2024-01-15T14:35:13.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.079+0000 7fe5382be700 1 -- start start 2024-01-15T14:35:13.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.080+0000 7fe5382be700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5300b3b50 0x7fe5300b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe5382be700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe530164a90 0x7fe53016bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe5382be700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53006ced0 con 0x7fe53006d4d0 2024-01-15T14:35:13.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe5382be700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53006da70 con 0x7fe5300b3b50 2024-01-15T14:35:13.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe5382be700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5300b4470 con 0x7fe530164a90 2024-01-15T14:35:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.080+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800ccf0 con 0x7f7d18005210 2024-01-15T14:35:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.080+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2800ce70 con 0x7f7d18005210 2024-01-15T14:35:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.080+0000 7f7d2d59a700 1 -- 172.21.15.77:0/2256866965 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28003c20 con 0x7f7d18005210 2024-01-15T14:35:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7f7d349b8700 1 -- 172.21.15.77:0/2256866965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d18005210 msgr2=0x7f7d1800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7f7d349b8700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d18005210 0x7f7d1800ba10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800c190 tx=0x7f7d18005880 comp rx=0 tx=0).stop 2024-01-15T14:35:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5300b3b50 0x7fe5300b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5300b3b50 0x7fe5300b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57656/0 (socket says 172.21.15.77:57656) 2024-01-15T14:35:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 -- 172.21.15.77:0/2197799670 learned_addr learned my addr 172.21.15.77:0/2197799670 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7f7d349b8700 1 -- 172.21.15.77:0/2256866965 shutdown_connections 2024-01-15T14:35:13.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7f7d349b8700 1 --2- 172.21.15.77:0/2256866965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d18005210 0x7f7d1800ba10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7f7d349b8700 1 -- 172.21.15.77:0/2256866965 >> 172.21.15.77:0/2256866965 conn(0x7f7d300cfa30 msgr2=0x7f7d300cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 -- 172.21.15.77:0/2197799670 >> v2:172.21.15.159:3300/0 conn(0x7fe530164a90 msgr2=0x7fe53016bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:35:13.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 --2- 172.21.15.77:0/2197799670 >> v2:172.21.15.159:3300/0 conn(0x7fe530164a90 0x7fe53016bee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 -- 172.21.15.77:0/2197799670 >> v1:172.21.15.77:6789/0 conn(0x7fe53006d4d0 legacy=0x7fe53006eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.081+0000 7fe53605a700 1 -- 172.21.15.77:0/2197799670 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5301634e0 con 0x7fe5300b3b50 2024-01-15T14:35:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.082+0000 7f7d349b8700 1 -- 172.21.15.77:0/2256866965 shutdown_connections 2024-01-15T14:35:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.082+0000 7fe53605a700 1 --2- 172.21.15.77:0/2197799670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5300b3b50 0x7fe5300b3f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe5300cf9e0 tx=0x7fe52800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6242b0c287fa5bb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.082+0000 7f7d349b8700 1 -- 172.21.15.77:0/2256866965 wait complete. 2024-01-15T14:35:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.082+0000 7fe535058700 1 -- 172.21.15.77:0/2197799670 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800e070 con 0x7fe5300b3b50 2024-01-15T14:35:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.082+0000 7fe535058700 1 -- 172.21.15.77:0/2197799670 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528003a20 con 0x7fe5300b3b50 2024-01-15T14:35:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.082+0000 7fe535058700 1 -- 172.21.15.77:0/2197799670 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528003dd0 con 0x7fe5300b3b50 2024-01-15T14:35:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.083+0000 7fe5382be700 1 -- 172.21.15.77:0/2197799670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5300b3b50 msgr2=0x7fe5300b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.083+0000 7fe5382be700 1 --2- 172.21.15.77:0/2197799670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5300b3b50 0x7fe5300b3f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe5300cf9e0 tx=0x7fe52800b330 comp rx=0 tx=0).stop 2024-01-15T14:35:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.084+0000 7f7d349b8700 1 Processor -- start 2024-01-15T14:35:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.084+0000 7f7d349b8700 1 -- start start 2024-01-15T14:35:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d349b8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18005210 0x7f7d300a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d349b8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3014fe90 0x7f7d301f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d349b8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 0x7f7d3015a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d349b8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3014d520 con 0x7f7d3015a470 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d349b8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3014c9a0 con 0x7f7d3014fe90 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d349b8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3014cbf0 con 0x7f7d18005210 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d2ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 0x7f7d3015a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d2ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 0x7f7d3015a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57660/0 (socket says 172.21.15.77:57660) 2024-01-15T14:35:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/419423315 learned_addr learned my addr 172.21.15.77:0/419423315 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7fec431f5700 1 Processor -- start 2024-01-15T14:35:13.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.085+0000 7f7d2dd9b700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3014fe90 0x7f7d301f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2e59c700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18005210 0x7f7d300a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fe5382be700 1 -- 172.21.15.77:0/2197799670 shutdown_connections 2024-01-15T14:35:13.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fe5382be700 1 --2- 172.21.15.77:0/2197799670 >> v2:172.21.15.159:3300/0 conn(0x7fe530164a90 0x7fe53016bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fe5382be700 1 --2- 172.21.15.77:0/2197799670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5300b3b50 0x7fe5300b3f30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fe5382be700 1 -- 172.21.15.77:0/2197799670 >> 172.21.15.77:0/2197799670 conn(0x7fe5300cfa30 msgr2=0x7fe5300cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18005210 msgr2=0x7f7d300a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18005210 0x7f7d300a9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3014fe90 msgr2=0x7f7d301f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3014fe90 0x7f7d301f7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2ed9d700 1 -- 172.21.15.77:0/419423315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3014ced0 con 0x7f7d3015a470 2024-01-15T14:35:13.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fe5382be700 1 -- 172.21.15.77:0/2197799670 shutdown_connections 2024-01-15T14:35:13.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2e59c700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18005210 0x7f7d300a9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:13.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2dd9b700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3014fe90 0x7f7d301f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:13.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fec431f5700 1 -- start start 2024-01-15T14:35:13.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d2ed9d700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 0x7f7d3015a8f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d30072c40 tx=0x7f7d280014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280052d0 con 0x7f7d3015a470 2024-01-15T14:35:13.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28005430 con 0x7f7d3015a470 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280057e0 con 0x7f7d3015a470 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.087+0000 7fec431f5700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fec3c071be0 0x7fec3c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.087+0000 7fec431f5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3c15d970 con 0x7fec3c0b4260 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.087+0000 7fec431f5700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3c0723c0 con 0x7fec3c0d0520 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.087+0000 7fec431f5700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c072d60 con 0x7fec3c071be0 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.087+0000 7fec33fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fec3c0b4260 0x7fec3c0714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38696/0 (socket says 172.21.15.77:38696) 2024-01-15T14:35:13.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.087+0000 7fec33fff700 1 -- 172.21.15.77:0/2615237151 learned_addr learned my addr 172.21.15.77:0/2615237151 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:13.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3014d1b0 con 0x7f7d3015a470 2024-01-15T14:35:13.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3015ae30 con 0x7f7d3015a470 2024-01-15T14:35:13.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262237581 0 0) 0x7fec3c15d970 con 0x7fec3c0b4260 2024-01-15T14:35:13.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec240035b0 con 0x7fec3c0b4260 2024-01-15T14:35:13.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599419362 0 0) 0x7fec3c0723c0 con 0x7fec3c0d0520 2024-01-15T14:35:13.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3c15d970 con 0x7fec3c0d0520 2024-01-15T14:35:13.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec41792700 1 --2- 172.21.15.77:0/2615237151 >> v2:172.21.15.159:3300/0 conn(0x7fec3c071be0 0x7fec3c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.086+0000 7fe5382be700 1 -- 172.21.15.77:0/2197799670 wait complete. 2024-01-15T14:35:13.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fe5382be700 1 Processor -- start 2024-01-15T14:35:13.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562075207 0 0) 0x7fec3c15d970 con 0x7fec3c0d0520 2024-01-15T14:35:13.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 >> v2:172.21.15.159:3300/0 conn(0x7fec3c071be0 msgr2=0x7fec3c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 --2- 172.21.15.77:0/2615237151 >> v2:172.21.15.159:3300/0 conn(0x7fec3c071be0 0x7fec3c15d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 >> v1:172.21.15.77:6789/0 conn(0x7fec3c0b4260 legacy=0x7fec3c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c160ba0 con 0x7fec3c0d0520 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fec2c0015a0 con 0x7fec3c0d0520 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 >> v1:172.21.15.159:6789/0 conn(0x7fec3c0d0520 legacy=0x7fec3c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec24004c40 0x7fec24005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec240037f0 0x7fec24003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c072d60 con 0x7fec24004c40 2024-01-15T14:35:13.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2400c030 con 0x7fec240041b0 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2400c1f0 con 0x7fec240037f0 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fec41792700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec33fff700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec240037f0 0x7fec24003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec40f91700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec24004c40 0x7fec24005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec41792700 1 -- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec240037f0 msgr2=0x7fec24003c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec41792700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec240037f0 0x7fec24003c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec41792700 1 -- 172.21.15.77:0/2615237151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec24004c40 msgr2=0x7fec24005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec41792700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec24004c40 0x7fec24005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec41792700 1 -- 172.21.15.77:0/2615237151 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c160ba0 con 0x7fec240041b0 2024-01-15T14:35:13.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fec41792700 1 -- 172.21.15.77:0/2615237151 reap_dead start 2024-01-15T14:35:13.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d2801d480 con 0x7f7d3015a470 2024-01-15T14:35:13.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.088+0000 7fe5382be700 1 -- start start 2024-01-15T14:35:13.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7f35a29ea700 1 Processor -- start 2024-01-15T14:35:13.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fe5382be700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe53006d4d0 0x7fe5300ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fe5382be700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 0x7fe530070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fe5382be700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe530164a90 0x7fe530071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fe5382be700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe530143330 con 0x7fe530164a90 2024-01-15T14:35:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fe5382be700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301427b0 con 0x7fe5300b3b50 2024-01-15T14:35:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.089+0000 7fe5382be700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe530142a90 con 0x7fe53006d4d0 2024-01-15T14:35:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7f7d1effd700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d1006b350 0x7f7d1006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 0x7fe530070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 0x7fe530070f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:58066/0 (socket says 172.21.15.77:58066) 2024-01-15T14:35:13.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 -- 172.21.15.77:0/508643896 learned_addr learned my addr 172.21.15.77:0/508643896 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe53605a700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe53006d4d0 0x7fe5300ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d2808b150 con 0x7f7d3015a470 2024-01-15T14:35:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7f7d2e59c700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d1006b350 0x7f7d1006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 -- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe53006d4d0 msgr2=0x7fe5300ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe53006d4d0 0x7fe5300ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 -- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe530164a90 msgr2=0x7fe530071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe530164a90 0x7fe530071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7fe535859700 1 -- 172.21.15.77:0/508643896 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528009b30 con 0x7fe5300b3b50 2024-01-15T14:35:13.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f7d2e59c700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d1006b350 0x7f7d1006d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7d30064c70 tx=0x7f7d24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7fec41792700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec2400ba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec34009f90 tx=0x7fec24005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=308ec380974ff9c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7fe535859700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 0x7fe530070f10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00c650 tx=0x7fe52c00c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.090+0000 7f35a29ea700 1 -- start start 2024-01-15T14:35:13.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f35a29ea700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f359c15d6a0 0x7f359c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f35a29ea700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 0x7f359c0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f35a29ea700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c071e90 con 0x7f359c071a10 2024-01-15T14:35:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f35a29ea700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c072d60 con 0x7f359c15d6a0 2024-01-15T14:35:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f35a29ea700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c06de80 con 0x7f359c0b4090 2024-01-15T14:35:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f359b7fe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 0x7f359c0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f359b7fe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 0x7f359c0714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53950/0 (socket says 172.21.15.77:53950) 2024-01-15T14:35:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7f359b7fe700 1 -- 172.21.15.77:0/2285574651 learned_addr learned my addr 172.21.15.77:0/2285574651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34003a20 con 0x7fec240041b0 2024-01-15T14:35:13.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3400f040 con 0x7fec240041b0 2024-01-15T14:35:13.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.091+0000 7fec337fe700 1 -- 172.21.15.77:0/2615237151 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34003dc0 con 0x7fec240041b0 2024-01-15T14:35:13.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7fec431f5700 1 -- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 msgr2=0x7fec2400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7fec431f5700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec2400ba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec34009f90 tx=0x7fec24005880 comp rx=0 tx=0).stop 2024-01-15T14:35:13.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7fec431f5700 1 -- 172.21.15.77:0/2615237151 shutdown_connections 2024-01-15T14:35:13.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7fec431f5700 1 --2- 172.21.15.77:0/2615237151 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec2400ba10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7fec431f5700 1 -- 172.21.15.77:0/2615237151 >> 172.21.15.77:0/2615237151 conn(0x7fec3c0cfa30 msgr2=0x7fec3c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7f359bfff700 1 --2- 172.21.15.77:0/2285574651 >> v2:172.21.15.159:3300/0 conn(0x7f359c15d6a0 0x7f359c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7f359b7fe700 1 -- 172.21.15.77:0/2285574651 >> v2:172.21.15.159:3300/0 conn(0x7f359c15d6a0 msgr2=0x7f359c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7f359b7fe700 1 --2- 172.21.15.77:0/2285574651 >> v2:172.21.15.159:3300/0 conn(0x7f359c15d6a0 0x7f359c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7f359b7fe700 1 -- 172.21.15.77:0/2285574651 >> v1:172.21.15.77:6789/0 conn(0x7f359c071a10 legacy=0x7f359c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7f359b7fe700 1 -- 172.21.15.77:0/2285574651 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0720e0 con 0x7f359c0b4090 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.092+0000 7f359b7fe700 1 --2- 172.21.15.77:0/2285574651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 0x7f359c0714d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f358c0088f0 tx=0x7f358c00d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d67b36cf0ccbf8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.093+0000 7f359affd700 1 -- 172.21.15.77:0/2285574651 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c010070 con 0x7f359c0b4090 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.093+0000 7f359affd700 1 -- 172.21.15.77:0/2285574651 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c00da40 con 0x7f359c0b4090 2024-01-15T14:35:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.093+0000 7f359affd700 1 -- 172.21.15.77:0/2285574651 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00ddb0 con 0x7f359c0b4090 2024-01-15T14:35:13.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.093+0000 7fec431f5700 1 -- 172.21.15.77:0/2615237151 shutdown_connections 2024-01-15T14:35:13.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.093+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d300727a0 con 0x7f7d3015a470 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7f35a29ea700 1 -- 172.21.15.77:0/2285574651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 msgr2=0x7f359c0714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7f35a29ea700 1 --2- 172.21.15.77:0/2285574651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 0x7f359c0714d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f358c0088f0 tx=0x7f358c00d330 comp rx=0 tx=0).stop 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7f35a29ea700 1 -- 172.21.15.77:0/2285574651 shutdown_connections 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7f35a29ea700 1 --2- 172.21.15.77:0/2285574651 >> v2:172.21.15.159:3300/0 conn(0x7f359c15d6a0 0x7f359c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7f35a29ea700 1 --2- 172.21.15.77:0/2285574651 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c0b4090 0x7f359c0714d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7f35a29ea700 1 -- 172.21.15.77:0/2285574651 >> 172.21.15.77:0/2285574651 conn(0x7f359c0cfa30 msgr2=0x7f359c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.095+0000 7f35a29ea700 1 -- 172.21.15.77:0/2285574651 shutdown_connections 2024-01-15T14:35:13.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7fec431f5700 1 -- 172.21.15.77:0/2615237151 wait complete. 2024-01-15T14:35:13.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.095+0000 7fec431f5700 1 Processor -- start 2024-01-15T14:35:13.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.095+0000 7f35a29ea700 1 -- 172.21.15.77:0/2285574651 wait complete. 2024-01-15T14:35:13.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7f35a29ea700 1 Processor -- start 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.096+0000 7fec431f5700 1 -- start start 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec431f5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec3c155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec431f5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 0x7fec3c156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec431f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec3c0ce980 0x7fec3c13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec431f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c0cf600 con 0x7fec3c0ce980 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec431f5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c156db0 con 0x7fec240041b0 2024-01-15T14:35:13.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec431f5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1557d0 con 0x7fec3c15c9d0 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec33fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 0x7fec3c156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec33fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 0x7fec3c156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53960/0 (socket says 172.21.15.77:53960) 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fec33fff700 1 -- 172.21.15.77:0/3262674598 learned_addr learned my addr 172.21.15.77:0/3262674598 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.098+0000 7fec33fff700 1 -- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 msgr2=0x7fec3c155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.098+0000 7fec33fff700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec3c155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.098+0000 7fec33fff700 1 -- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec3c0ce980 msgr2=0x7fec3c13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.098+0000 7fec33fff700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec3c0ce980 0x7fec3c13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.098+0000 7fec33fff700 1 -- 172.21.15.77:0/3262674598 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c155ab0 con 0x7fec3c15c9d0 2024-01-15T14:35:13.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d28056100 con 0x7f7d3015a470 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c0054e0 con 0x7fe5300b3b50 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c003680 con 0x7fe5300b3b50 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.094+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c003a30 con 0x7fe5300b3b50 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe530142d70 con 0x7fe5300b3b50 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe530142380 con 0x7fe5300b3b50 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53014b7a0 con 0x7fe5300b3b50 2024-01-15T14:35:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe52c003b90 con 0x7fe5300b3b50 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.097+0000 7f35a29ea700 1 -- start start 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a29ea700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c071a10 0x7f359c157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a29ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f359c0b4090 0x7f359c1fbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a29ea700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 0x7f359c13fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a29ea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1f8160 con 0x7f359c0b4090 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a29ea700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1f8440 con 0x7f359c15d6a0 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a29ea700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1f8720 con 0x7f359c071a10 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a0f87700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 0x7f359c13fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a0f87700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 0x7f359c13fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:58090/0 (socket says 172.21.15.77:58090) 2024-01-15T14:35:13.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f35a0f87700 1 -- 172.21.15.77:0/29210945 learned_addr learned my addr 172.21.15.77:0/29210945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.099+0000 7f359bfff700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c071a10 0x7f359c157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f35a0f87700 1 -- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c071a10 msgr2=0x7f359c157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f35a0f87700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c071a10 0x7f359c157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f35a0f87700 1 -- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f359c0b4090 msgr2=0x7f359c1fbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f35a0f87700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f359c0b4090 0x7f359c1fbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f35a0f87700 1 -- 172.21.15.77:0/29210945 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0085d0 con 0x7f359c15d6a0 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f35a0f87700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 0x7f359c13fd80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f359400c7a0 tx=0x7f359400cab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594003e10 con 0x7f359c15d6a0 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594005070 con 0x7f359c15d6a0 2024-01-15T14:35:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.100+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594005420 con 0x7f359c15d6a0 2024-01-15T14:35:13.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.101+0000 7fec33fff700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 0x7fec3c156870 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fec38000f60 tx=0x7fec3800db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.101+0000 7fe526ffd700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe51406b3a0 0x7fe51406d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.101+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe52c0890d0 con 0x7fe5300b3b50 2024-01-15T14:35:13.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.101+0000 7fe53605a700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe51406b3a0 0x7fe51406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.102+0000 7fe53605a700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe51406b3a0 0x7fe51406d860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe52800b370 tx=0x7fe528009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.101+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c1f8a00 con 0x7f359c15d6a0 2024-01-15T14:35:13.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.101+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c158350 con 0x7f359c15d6a0 2024-01-15T14:35:13.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.102+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c0751e0 con 0x7f359c15d6a0 2024-01-15T14:35:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.105+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe52c053fb0 con 0x7fe5300b3b50 2024-01-15T14:35:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.106+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38005d70 con 0x7fec3c15c9d0 2024-01-15T14:35:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.106+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec380070f0 con 0x7fec3c15c9d0 2024-01-15T14:35:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.106+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38007480 con 0x7fec3c15c9d0 2024-01-15T14:35:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.107+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c155d90 con 0x7fec3c15c9d0 2024-01-15T14:35:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.107+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c156350 con 0x7fec3c15c9d0 2024-01-15T14:35:13.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.109+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c0052f0 con 0x7fec3c15c9d0 2024-01-15T14:35:13.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.107+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3594006ba0 con 0x7f359c15d6a0 2024-01-15T14:35:13.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.108+0000 7f3598ff9700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f358406b500 0x7f358406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.108+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3594088fe0 con 0x7f359c15d6a0 2024-01-15T14:35:13.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.110+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3594053e70 con 0x7f359c15d6a0 2024-01-15T14:35:13.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.111+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fec380075e0 con 0x7fec3c15c9d0 2024-01-15T14:35:13.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.112+0000 7fec317fa700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fec2806b3a0 0x7fec2806d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.112+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec38089940 con 0x7fec3c15c9d0 2024-01-15T14:35:13.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.114+0000 7fec40f91700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fec2806b3a0 0x7fec2806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.114+0000 7fec40f91700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fec2806b3a0 0x7fec2806d860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fec2c001bc0 tx=0x7fec2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.117+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec38054820 con 0x7fec3c15c9d0 2024-01-15T14:35:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.121+0000 7f359bfff700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f358406b500 0x7f358406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.122+0000 7f359bfff700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f358406b500 0x7f358406d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f359c157da0 tx=0x7f3590008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.129+0000 7fbf34f98700 1 Processor -- start 2024-01-15T14:35:13.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.129+0000 7fbf34f98700 1 -- start start 2024-01-15T14:35:13.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.131+0000 7fbf34f98700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf280df870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.131+0000 7fbf34f98700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28005ef0 con 0x7fbf28055640 2024-01-15T14:35:13.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.131+0000 7fbf34f98700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2804f890 con 0x7fbf28050290 2024-01-15T14:35:13.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.131+0000 7fbf34f98700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280539e0 con 0x7fbf280dcfb0 2024-01-15T14:35:13.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.132+0000 7fbf2ffff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf280df870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.132+0000 7fbf2ffff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf280df870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:53966/0 (socket says 172.21.15.77:53966) 2024-01-15T14:35:13.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.132+0000 7fbf2ffff700 1 -- 172.21.15.77:0/2028832142 learned_addr learned my addr 172.21.15.77:0/2028832142 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.132+0000 7fbf2ffff700 1 -- 172.21.15.77:0/2028832142 >> v1:172.21.15.159:6789/0 conn(0x7fbf28050290 legacy=0x7fbf28006380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.132+0000 7fbf2ffff700 1 -- 172.21.15.77:0/2028832142 >> v1:172.21.15.77:6789/0 conn(0x7fbf28055640 legacy=0x7fbf2804fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.132+0000 7fbf2ffff700 1 -- 172.21.15.77:0/2028832142 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28053cc0 con 0x7fbf280dcfb0 2024-01-15T14:35:13.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.133+0000 7fbf2ffff700 1 --2- 172.21.15.77:0/2028832142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf280df870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009920 tx=0x7fbf24009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8582dfa21046df52 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.133+0000 7fbf2e7fc700 1 -- 172.21.15.77:0/2028832142 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004230 con 0x7fbf280dcfb0 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.133+0000 7fbf2e7fc700 1 -- 172.21.15.77:0/2028832142 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf24004390 con 0x7fbf280dcfb0 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.133+0000 7fbf2e7fc700 1 -- 172.21.15.77:0/2028832142 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24009e70 con 0x7fbf280dcfb0 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.134+0000 7fbf34f98700 1 -- 172.21.15.77:0/2028832142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 msgr2=0x7fbf280df870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.134+0000 7fbf34f98700 1 --2- 172.21.15.77:0/2028832142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf280df870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009920 tx=0x7fbf24009c30 comp rx=0 tx=0).stop 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.135+0000 7fbf34f98700 1 -- 172.21.15.77:0/2028832142 shutdown_connections 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.135+0000 7fbf34f98700 1 --2- 172.21.15.77:0/2028832142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf280df870 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.135+0000 7fbf34f98700 1 -- 172.21.15.77:0/2028832142 >> 172.21.15.77:0/2028832142 conn(0x7fbf28054b20 msgr2=0x7fbf28054f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.135+0000 7fbf34f98700 1 -- 172.21.15.77:0/2028832142 shutdown_connections 2024-01-15T14:35:13.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.135+0000 7fbf34f98700 1 -- 172.21.15.77:0/2028832142 wait complete. 2024-01-15T14:35:13.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.137+0000 7fbf34f98700 1 Processor -- start 2024-01-15T14:35:13.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.137+0000 7fbf34f98700 1 -- start start 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf34f98700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf28050290 0x7fbf2804d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf34f98700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf28055640 0x7fbf280deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf34f98700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf281889d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf34f98700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280ce910 con 0x7fbf28050290 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf34f98700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28189080 con 0x7fbf28055640 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf34f98700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28189320 con 0x7fbf280dcfb0 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf2f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf28050290 0x7fbf2804d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf2f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf28050290 0x7fbf2804d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57702/0 (socket says 172.21.15.77:57702) 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf2f7fe700 1 -- 172.21.15.77:0/860752984 learned_addr learned my addr 172.21.15.77:0/860752984 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.139+0000 7fbf2effd700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf28055640 0x7fbf280deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.140+0000 7fbf2effd700 1 -- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 msgr2=0x7fbf281889d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.140+0000 7fbf2effd700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf281889d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.140+0000 7fbf2effd700 1 -- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf28050290 msgr2=0x7fbf2804d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.140+0000 7fbf2effd700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf28050290 0x7fbf2804d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.140+0000 7fbf2effd700 1 -- 172.21.15.77:0/860752984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24009550 con 0x7fbf28055640 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.141+0000 7fbf2effd700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf28055640 0x7fbf280deaa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf280086e0 tx=0x7fbf1c00c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.141+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0052d0 con 0x7fbf28055640 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.141+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c005430 con 0x7fbf28055640 2024-01-15T14:35:13.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.141+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c003880 con 0x7fbf28055640 2024-01-15T14:35:13.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.141+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf28189600 con 0x7fbf28055640 2024-01-15T14:35:13.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.141+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf28189bc0 con 0x7fbf28055640 2024-01-15T14:35:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.145+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbf1c0039e0 con 0x7fbf28055640 2024-01-15T14:35:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.146+0000 7fbf1bfff700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbf1406b420 0x7fbf1406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.146+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf1c088d90 con 0x7fbf28055640 2024-01-15T14:35:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.147+0000 7fbf2f7fe700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbf1406b420 0x7fbf1406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.147+0000 7fbf2f7fe700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbf1406b420 0x7fbf1406d8e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf3007aef0 tx=0x7fbf3008bf90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.148+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf100052f0 con 0x7fbf28055640 2024-01-15T14:35:13.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.153+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf1c053c90 con 0x7fbf28055640 2024-01-15T14:35:13.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.342+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7feee00055a0 con 0x7feefc06c710 2024-01-15T14:35:13.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.343+0000 7feeeaffd700 1 -- 172.21.15.77:0/1724450789 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feeec059590 con 0x7feefc06c710 2024-01-15T14:35:13.347 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7feedc06b450 msgr2=0x7feedc06d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7feedc06b450 0x7feedc06d910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feefc155bb0 tx=0x7feef0006c00 comp rx=0 tx=0).stop 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 msgr2=0x7feefc157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 0x7feefc157500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feeec009fd0 tx=0x7feeec004750 comp rx=0 tx=0).stop 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 shutdown_connections 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7feedc06b450 0x7feedc06d910 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7feefc150330 0x7feefc155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feefc06c710 0x7feefc157500 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.347+0000 7fef00a3d700 1 --2- 172.21.15.77:0/1724450789 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7feefc15e080 0x7feefc200cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.348+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 >> 172.21.15.77:0/1724450789 conn(0x7feefc071b90 msgr2=0x7feefc0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.348+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 shutdown_connections 2024-01-15T14:35:13.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.348+0000 7fef00a3d700 1 -- 172.21.15.77:0/1724450789 wait complete. 2024-01-15T14:35:13.367 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.1 2024-01-15T14:35:13.368 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.412+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8ca81f4f70 con 0x7f8ca81603d0 2024-01-15T14:35:13.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.413+0000 7f8c9e7fc700 1 -- 172.21.15.77:0/877640140 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ca81f4f70 con 0x7f8ca81603d0 2024-01-15T14:35:13.415 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-01-15T14:35:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c9406b500 msgr2=0x7f8c9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c9406b500 0x7f8c9406d9c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8138c60 tx=0x7f8ca0001930 comp rx=0 tx=0).stop 2024-01-15T14:35:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 msgr2=0x7f8ca813bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca813bab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800e810 tx=0x7f8c9800ebd0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 shutdown_connections 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c9406b500 0x7f8c9406d9c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8ca8145740 0x7f8ca8138210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ca81603d0 0x7f8ca813bab0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 --2- 172.21.15.77:0/877640140 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8ca815fd70 0x7f8ca813b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 >> 172.21.15.77:0/877640140 conn(0x7f8ca80b1740 msgr2=0x7f8ca80b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 shutdown_connections 2024-01-15T14:35:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.417+0000 7f8cafe66700 1 -- 172.21.15.77:0/877640140 wait complete. 2024-01-15T14:35:13.437 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.6 2024-01-15T14:35:13.437 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.456+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6ca00055a0 con 0x7f6cc015d6a0 2024-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.457+0000 7f6cb5ffb700 1 -- 172.21.15.77:0/201247842 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6cb8006370 con 0x7f6cc015d6a0 2024-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6ca406b350 msgr2=0x7f6ca406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6ca406b350 0x7f6ca406d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc008010 tx=0x7f6cbc01a040 comp rx=0 tx=0).stop 2024-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cc015d6a0 msgr2=0x7f6cc013cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cc015d6a0 0x7f6cc013cfe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8001a20 tx=0x7f6cb8001d30 comp rx=0 tx=0).stop 2024-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 shutdown_connections 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6ca406b350 0x7f6ca406d810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6cc015d6a0 0x7f6cc013cfe0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc0071be0 0x7f6cc015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 --2- 172.21.15.77:0/201247842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6cc00b4090 0x7f6cc01f80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 >> 172.21.15.77:0/201247842 conn(0x7f6cc00cfa30 msgr2=0x7f6cc01402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 shutdown_connections 2024-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.461+0000 7f6cc72aa700 1 -- 172.21.15.77:0/201247842 wait complete. 2024-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.466+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbf100055a0 con 0x7fbf28055640 2024-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.467+0000 7fbf1bfff700 1 -- 172.21.15.77:0/860752984 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbf1c017030 con 0x7fbf28055640 2024-01-15T14:35:13.470 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbf1406b420 msgr2=0x7fbf1406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbf1406b420 0x7fbf1406d8e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf3007aef0 tx=0x7fbf3008bf90 comp rx=0 tx=0).stop 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf28055640 msgr2=0x7fbf280deaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf28055640 0x7fbf280deaa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf280086e0 tx=0x7fbf1c00c7b0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 shutdown_connections 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbf1406b420 0x7fbf1406d8e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbf280dcfb0 0x7fbf281889d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf28050290 0x7fbf2804d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 --2- 172.21.15.77:0/860752984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbf28055640 0x7fbf280deaa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.469+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 >> 172.21.15.77:0/860752984 conn(0x7fbf28054b20 msgr2=0x7fbf280071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.470+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 shutdown_connections 2024-01-15T14:35:13.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.470+0000 7fbf34f98700 1 -- 172.21.15.77:0/860752984 wait complete. 2024-01-15T14:35:13.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.2 2024-01-15T14:35:13.481 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.0 2024-01-15T14:35:13.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.482+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fec1c0055a0 con 0x7fec3c15c9d0 2024-01-15T14:35:13.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.483+0000 7fec317fa700 1 -- 172.21.15.77:0/3262674598 <== mon.2 v2:172.21.15.159:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec38057e40 con 0x7fec3c15c9d0 2024-01-15T14:35:13.485 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.485 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-01-15T14:35:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.485+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fec2806b3a0 msgr2=0x7fec2806d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.485+0000 7fec431f5700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fec2806b3a0 0x7fec2806d860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fec2c001bc0 tx=0x7fec2c009040 comp rx=0 tx=0).stop 2024-01-15T14:35:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.485+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 msgr2=0x7fec3c156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.485+0000 7fec431f5700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 0x7fec3c156870 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fec38000f60 tx=0x7fec3800db00 comp rx=0 tx=0).stop 2024-01-15T14:35:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 shutdown_connections 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fec2806b3a0 0x7fec2806d860 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec3c0ce980 0x7fec3c13fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fec240041b0 0x7fec3c155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 --2- 172.21.15.77:0/3262674598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fec3c15c9d0 0x7fec3c156870 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 >> 172.21.15.77:0/3262674598 conn(0x7fec3c0cfa30 msgr2=0x7fec3c141300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 shutdown_connections 2024-01-15T14:35:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.486+0000 7fec431f5700 1 -- 172.21.15.77:0/3262674598 wait complete. 2024-01-15T14:35:13.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.5 2024-01-15T14:35:13.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.498+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe5300751e0 con 0x7fe5300b3b50 2024-01-15T14:35:13.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.498+0000 7fe526ffd700 1 -- 172.21.15.77:0/508643896 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe52c017020 con 0x7fe5300b3b50 2024-01-15T14:35:13.500 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.500 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:13.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe51406b3a0 msgr2=0x7fe51406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe51406b3a0 0x7fe51406d860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe52800b370 tx=0x7fe528009000 comp rx=0 tx=0).stop 2024-01-15T14:35:13.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 msgr2=0x7fe530070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 0x7fe530070f10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00c650 tx=0x7fe52c00c960 comp rx=0 tx=0).stop 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 shutdown_connections 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe51406b3a0 0x7fe51406d860 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe53006d4d0 0x7fe5300ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe530164a90 0x7fe530071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 --2- 172.21.15.77:0/508643896 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe5300b3b50 0x7fe530070f10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.500+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 >> 172.21.15.77:0/508643896 conn(0x7fe5300cfa30 msgr2=0x7fe5300b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.501+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 shutdown_connections 2024-01-15T14:35:13.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.501+0000 7fe5382be700 1 -- 172.21.15.77:0/508643896 wait complete. 2024-01-15T14:35:13.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.502+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7d3014d340 con 0x7f7d3015a470 2024-01-15T14:35:13.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.503+0000 7f7d1effd700 1 -- 172.21.15.77:0/419423315 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d28016020 con 0x7f7d3015a470 2024-01-15T14:35:13.505 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-01-15T14:35:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d1006b350 msgr2=0x7f7d1006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d1006b350 0x7f7d1006d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7d30064c70 tx=0x7f7d24008040 comp rx=0 tx=0).stop 2024-01-15T14:35:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 msgr2=0x7f7d3015a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 0x7f7d3015a8f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d30072c40 tx=0x7f7d280014a0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 shutdown_connections 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d1006b350 0x7f7d1006d810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d18005210 0x7f7d300a9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3015a470 0x7f7d3015a8f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 --2- 172.21.15.77:0/419423315 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3014fe90 0x7f7d301f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.506+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 >> 172.21.15.77:0/419423315 conn(0x7f7d300cfa30 msgr2=0x7f7d300ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.507+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 shutdown_connections 2024-01-15T14:35:13.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.508+0000 7f7d349b8700 1 -- 172.21.15.77:0/419423315 wait complete. 2024-01-15T14:35:13.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.3 2024-01-15T14:35:13.516 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.516+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f359c1f8580 con 0x7f359c15d6a0 2024-01-15T14:35:13.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.516+0000 7f3598ff9700 1 -- 172.21.15.77:0/29210945 <== mon.1 v2:172.21.15.159:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3594017020 con 0x7f359c15d6a0 2024-01-15T14:35:13.517 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2024-01-15T14:35:13.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f358406b500 msgr2=0x7f358406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f358406b500 0x7f358406d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f359c157da0 tx=0x7f3590008040 comp rx=0 tx=0).stop 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 msgr2=0x7f359c13fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 0x7f359c13fd80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f359400c7a0 tx=0x7f359400cab0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 shutdown_connections 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f358406b500 0x7f358406d9c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f359c071a10 0x7f359c157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f359c15d6a0 0x7f359c13fd80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 --2- 172.21.15.77:0/29210945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f359c0b4090 0x7f359c1fbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 >> 172.21.15.77:0/29210945 conn(0x7f359c0cfa30 msgr2=0x7f359c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 shutdown_connections 2024-01-15T14:35:13.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.519+0000 7f35a29ea700 1 -- 172.21.15.77:0/29210945 wait complete. 2024-01-15T14:35:13.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.7 2024-01-15T14:35:13.523 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.4 2024-01-15T14:35:13.533 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:13.533 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T14:35:13.533 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T14:35:13.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.694+0000 7f7c71efb700 1 Processor -- start 2024-01-15T14:35:13.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.694+0000 7f7c71efb700 1 -- start start 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c71efb700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7c6c15e080 0x7f7c6c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c71efb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c0af4d0 0x7f7c6c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c71efb700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6c0ae430 con 0x7f7c6c06c000 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c71efb700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6c0af0b0 con 0x7f7c6c15e080 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c71efb700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c0b05a0 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c0af4d0 0x7f7c6c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6b7fe700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7c6c15e080 0x7f7c6c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6bfff700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7c6c06c000 0x7f7c6c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:35770/0 (socket says 172.21.15.77:35770) 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6bfff700 1 -- 172.21.15.77:0/1689883027 learned_addr learned my addr 172.21.15.77:0/1689883027 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6affd700 1 -- 172.21.15.77:0/1689883027 >> v2:172.21.15.159:3300/0 conn(0x7f7c6c15e080 msgr2=0x7f7c6c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6affd700 1 --2- 172.21.15.77:0/1689883027 >> v2:172.21.15.159:3300/0 conn(0x7f7c6c15e080 0x7f7c6c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6affd700 1 -- 172.21.15.77:0/1689883027 >> v1:172.21.15.159:6789/0 conn(0x7f7c6c06c000 legacy=0x7f7c6c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.695+0000 7f7c6affd700 1 -- 172.21.15.77:0/1689883027 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6c0ae710 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c6b7fe700 1 --2- 172.21.15.77:0/1689883027 >> v2:172.21.15.159:3300/0 conn(0x7f7c6c15e080 0x7f7c6c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c6affd700 1 --2- 172.21.15.77:0/1689883027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c0af4d0 0x7f7c6c0afb90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009840 tx=0x7f7c60009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d40b8f473b1d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c6a7fc700 1 -- 172.21.15.77:0/1689883027 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60011830 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c6a7fc700 1 -- 172.21.15.77:0/1689883027 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c60011990 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c6a7fc700 1 -- 172.21.15.77:0/1689883027 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60011d20 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 -- 172.21.15.77:0/1689883027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c0af4d0 msgr2=0x7f7c6c0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 --2- 172.21.15.77:0/1689883027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c0af4d0 0x7f7c6c0afb90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009840 tx=0x7f7c60009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 -- 172.21.15.77:0/1689883027 shutdown_connections 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 --2- 172.21.15.77:0/1689883027 >> v2:172.21.15.159:3300/0 conn(0x7f7c6c15e080 0x7f7c6c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 --2- 172.21.15.77:0/1689883027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c0af4d0 0x7f7c6c0afb90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 -- 172.21.15.77:0/1689883027 >> 172.21.15.77:0/1689883027 conn(0x7f7c6c071b90 msgr2=0x7f7c6c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 -- 172.21.15.77:0/1689883027 shutdown_connections 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.696+0000 7f7c71efb700 1 -- 172.21.15.77:0/1689883027 wait complete. 2024-01-15T14:35:13.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.698+0000 7f7c71efb700 1 Processor -- start 2024-01-15T14:35:13.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.698+0000 7f7c71efb700 1 -- start start 2024-01-15T14:35:13.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c6c06c000 0x7f7c6c1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 0x7f7c6c160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c15e080 0x7f7c6c160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c15f150 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c15f430 con 0x7f7c6c06c000 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c15f710 con 0x7f7c6c15e080 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 0x7f7c6c160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 0x7f7c6c160770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57706/0 (socket says 172.21.15.77:57706) 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 -- 172.21.15.77:0/721444059 learned_addr learned my addr 172.21.15.77:0/721444059 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6bfff700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c15e080 0x7f7c6c160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6b7fe700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c6c06c000 0x7f7c6c1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 -- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c15e080 msgr2=0x7f7c6c160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c15e080 0x7f7c6c160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 -- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c6c06c000 msgr2=0x7f7c6c1519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c6c06c000 0x7f7c6c1519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 -- 172.21.15.77:0/721444059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c0094a0 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c6affd700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 0x7f7c6c160770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009fd0 tx=0x7f7c60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c600038a0 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6001f070 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c600094a0 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60003c40 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.699+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c158790 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.700+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6c0753f0 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.701+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7c60026480 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.701+0000 7f7c53fff700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7c5806b350 0x7f7c5806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.701+0000 7f7c6b7fe700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7c5806b350 0x7f7c5806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.701+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c60021070 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.702+0000 7f7c6b7fe700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7c5806b350 0x7f7c5806d810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009c60 tx=0x7f7c5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.706+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c60056390 con 0x7f7c6c0af4d0 2024-01-15T14:35:13.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.895+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c6c0aecd0 con 0x7f7c5806b350 2024-01-15T14:35:13.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.902+0000 7f7c53fff700 1 -- 172.21.15.77:0/721444059 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264572 (secure 0 0 0) 0x7f7c5c009da0 con 0x7f7c5806b350 2024-01-15T14:35:13.909 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:13.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7c5806b350 msgr2=0x7f7c5806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7c5806b350 0x7f7c5806d810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009c60 tx=0x7f7c5c008040 comp rx=0 tx=0).stop 2024-01-15T14:35:13.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 msgr2=0x7f7c6c160770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 0x7f7c6c160770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009fd0 tx=0x7f7c60004750 comp rx=0 tx=0).stop 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 shutdown_connections 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7c5806b350 0x7f7c5806d810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7c6c15e080 0x7f7c6c160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7c6c06c000 0x7f7c6c1519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 --2- 172.21.15.77:0/721444059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c0af4d0 0x7f7c6c160770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 >> 172.21.15.77:0/721444059 conn(0x7f7c6c071b90 msgr2=0x7f7c6c0b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 shutdown_connections 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:13.905+0000 7f7c71efb700 1 -- 172.21.15.77:0/721444059 wait complete. 2024-01-15T14:35:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-01-15T14:35:13.918 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-15T14:35:13.021067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003301"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223558+0000","last_change":"2024-01-15T14:35:05.242883+0000","last_active":"2024-01-15T14:35:06.223558+0000","last_peered":"2024-01-15T14:35:06.223558+0000","last_clean":"2024-01-15T14:35:06.223558+0000","last_became_active":"2024-01-15T14:35:05.242650+0000","last_became_peered":"2024-01-15T14:35:05.242650+0000","last_unstale":"2024-01-15T14:35:06.223558+0000","last_undegraded":"2024-01-15T14:35:06.223558+0000","last_fullsized":"2024-01-15T14:35:06.223558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:06:13.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038751+0000","last_change":"2024-01-15T14:35:05.230934+0000","last_active":"2024-01-15T14:35:07.038751+0000","last_peered":"2024-01-15T14:35:07.038751+0000","last_clean":"2024-01-15T14:35:07.038751+0000","last_became_active":"2024-01-15T14:35:05.230356+0000","last_became_peered":"2024-01-15T14:35:05.230356+0000","last_unstale":"2024-01-15T14:35:07.038751+0000","last_undegraded":"2024-01-15T14:35:07.038751+0000","last_fullsized":"2024-01-15T14:35:07.038751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914251+0000","last_change":"2024-01-15T14:35:05.228307+0000","last_active":"2024-01-15T14:35:06.914251+0000","last_peered":"2024-01-15T14:35:06.914251+0000","last_clean":"2024-01-15T14:35:06.914251+0000","last_became_active":"2024-01-15T14:35:05.227876+0000","last_became_peered":"2024-01-15T14:35:05.227876+0000","last_unstale":"2024-01-15T14:35:06.914251+0000","last_undegraded":"2024-01-15T14:35:06.914251+0000","last_fullsized":"2024-01-15T14:35:06.914251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:11.269770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338166+0000","last_change":"2024-01-15T14:35:05.236667+0000","last_active":"2024-01-15T14:35:06.338166+0000","last_peered":"2024-01-15T14:35:06.338166+0000","last_clean":"2024-01-15T14:35:06.338166+0000","last_became_active":"2024-01-15T14:35:05.236363+0000","last_became_peered":"2024-01-15T14:35:05.236363+0000","last_unstale":"2024-01-15T14:35:06.338166+0000","last_undegraded":"2024-01-15T14:35:06.338166+0000","last_fullsized":"2024-01-15T14:35:06.338166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222610+0000","last_change":"2024-01-15T14:35:05.239721+0000","last_active":"2024-01-15T14:35:06.222610+0000","last_peered":"2024-01-15T14:35:06.222610+0000","last_clean":"2024-01-15T14:35:06.222610+0000","last_became_active":"2024-01-15T14:35:05.235648+0000","last_became_peered":"2024-01-15T14:35:05.235648+0000","last_unstale":"2024-01-15T14:35:06.222610+0000","last_undegraded":"2024-01-15T14:35:06.222610+0000","last_fullsized":"2024-01-15T14:35:06.222610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337634+0000","last_change":"2024-01-15T14:35:05.230388+0000","last_active":"2024-01-15T14:35:06.337634+0000","last_peered":"2024-01-15T14:35:06.337634+0000","last_clean":"2024-01-15T14:35:06.337634+0000","last_became_active":"2024-01-15T14:35:05.230109+0000","last_became_peered":"2024-01-15T14:35:05.230109+0000","last_unstale":"2024-01-15T14:35:06.337634+0000","last_undegraded":"2024-01-15T14:35:06.337634+0000","last_fullsized":"2024-01-15T14:35:06.337634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222611+0000","last_change":"2024-01-15T14:35:05.239457+0000","last_active":"2024-01-15T14:35:06.222611+0000","last_peered":"2024-01-15T14:35:06.222611+0000","last_clean":"2024-01-15T14:35:06.222611+0000","last_became_active":"2024-01-15T14:35:05.239152+0000","last_became_peered":"2024-01-15T14:35:05.239152+0000","last_unstale":"2024-01-15T14:35:06.222611+0000","last_undegraded":"2024-01-15T14:35:06.222611+0000","last_fullsized":"2024-01-15T14:35:06.222611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220500+0000","last_change":"2024-01-15T14:35:05.219905+0000","last_active":"2024-01-15T14:35:06.220500+0000","last_peered":"2024-01-15T14:35:06.220500+0000","last_clean":"2024-01-15T14:35:06.220500+0000","last_became_active":"2024-01-15T14:35:05.219331+0000","last_became_peered":"2024-01-15T14:35:05.219331+0000","last_unstale":"2024-01-15T14:35:06.220500+0000","last_undegraded":"2024-01-15T14:35:06.220500+0000","last_fullsized":"2024-01-15T14:35:06.220500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223409+0000","last_change":"2024-01-15T14:35:05.232665+0000","last_active":"2024-01-15T14:35:06.223409+0000","last_peered":"2024-01-15T14:35:06.223409+0000","last_clean":"2024-01-15T14:35:06.223409+0000","last_became_active":"2024-01-15T14:35:05.232381+0000","last_became_peered":"2024-01-15T14:35:05.232381+0000","last_unstale":"2024-01-15T14:35:06.223409+0000","last_undegraded":"2024-01-15T14:35:06.223409+0000","last_fullsized":"2024-01-15T14:35:06.223409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222267+0000","last_change":"2024-01-15T14:35:05.222078+0000","last_active":"2024-01-15T14:35:06.222267+0000","last_peered":"2024-01-15T14:35:06.222267+0000","last_clean":"2024-01-15T14:35:06.222267+0000","last_became_active":"2024-01-15T14:35:05.221809+0000","last_became_peered":"2024-01-15T14:35:05.221809+0000","last_unstale":"2024-01-15T14:35:06.222267+0000","last_undegraded":"2024-01-15T14:35:06.222267+0000","last_fullsized":"2024-01-15T14:35:06.222267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221969+0000","last_change":"2024-01-15T14:35:05.222838+0000","last_active":"2024-01-15T14:35:06.221969+0000","last_peered":"2024-01-15T14:35:06.221969+0000","last_clean":"2024-01-15T14:35:06.221969+0000","last_became_active":"2024-01-15T14:35:05.221973+0000","last_became_peered":"2024-01-15T14:35:05.221973+0000","last_unstale":"2024-01-15T14:35:06.221969+0000","last_undegraded":"2024-01-15T14:35:06.221969+0000","last_fullsized":"2024-01-15T14:35:06.221969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038638+0000","last_change":"2024-01-15T14:35:05.242599+0000","last_active":"2024-01-15T14:35:07.038638+0000","last_peered":"2024-01-15T14:35:07.038638+0000","last_clean":"2024-01-15T14:35:07.038638+0000","last_became_active":"2024-01-15T14:35:05.242171+0000","last_became_peered":"2024-01-15T14:35:05.242171+0000","last_unstale":"2024-01-15T14:35:07.038638+0000","last_undegraded":"2024-01-15T14:35:07.038638+0000","last_fullsized":"2024-01-15T14:35:07.038638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220701+0000","last_change":"2024-01-15T14:35:05.224493+0000","last_active":"2024-01-15T14:35:06.220701+0000","last_peered":"2024-01-15T14:35:06.220701+0000","last_clean":"2024-01-15T14:35:06.220701+0000","last_became_active":"2024-01-15T14:35:05.224166+0000","last_became_peered":"2024-01-15T14:35:05.224166+0000","last_unstale":"2024-01-15T14:35:06.220701+0000","last_undegraded":"2024-01-15T14:35:06.220701+0000","last_fullsized":"2024-01-15T14:35:06.220701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914368+0000","last_change":"2024-01-15T14:35:05.229113+0000","last_active":"2024-01-15T14:35:06.914368+0000","last_peered":"2024-01-15T14:35:06.914368+0000","last_clean":"2024-01-15T14:35:06.914368+0000","last_became_active":"2024-01-15T14:35:05.228790+0000","last_became_peered":"2024-01-15T14:35:05.228790+0000","last_unstale":"2024-01-15T14:35:06.914368+0000","last_undegraded":"2024-01-15T14:35:06.914368+0000","last_fullsized":"2024-01-15T14:35:06.914368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038480+0000","last_change":"2024-01-15T14:35:05.236987+0000","last_active":"2024-01-15T14:35:07.038480+0000","last_peered":"2024-01-15T14:35:07.038480+0000","last_clean":"2024-01-15T14:35:07.038480+0000","last_became_active":"2024-01-15T14:35:05.236693+0000","last_became_peered":"2024-01-15T14:35:05.236693+0000","last_unstale":"2024-01-15T14:35:07.038480+0000","last_undegraded":"2024-01-15T14:35:07.038480+0000","last_fullsized":"2024-01-15T14:35:07.038480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337512+0000","last_change":"2024-01-15T14:35:05.229665+0000","last_active":"2024-01-15T14:35:06.337512+0000","last_peered":"2024-01-15T14:35:06.337512+0000","last_clean":"2024-01-15T14:35:06.337512+0000","last_became_active":"2024-01-15T14:35:05.229391+0000","last_became_peered":"2024-01-15T14:35:05.229391+0000","last_unstale":"2024-01-15T14:35:06.337512+0000","last_undegraded":"2024-01-15T14:35:06.337512+0000","last_fullsized":"2024-01-15T14:35:06.337512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338230+0000","last_change":"2024-01-15T14:35:05.232892+0000","last_active":"2024-01-15T14:35:06.338230+0000","last_peered":"2024-01-15T14:35:06.338230+0000","last_clean":"2024-01-15T14:35:06.338230+0000","last_became_active":"2024-01-15T14:35:05.232596+0000","last_became_peered":"2024-01-15T14:35:05.232596+0000","last_unstale":"2024-01-15T14:35:06.338230+0000","last_undegraded":"2024-01-15T14:35:06.338230+0000","last_fullsized":"2024-01-15T14:35:06.338230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420132+0000","last_change":"2024-01-15T14:35:05.224599+0000","last_active":"2024-01-15T14:35:06.420132+0000","last_peered":"2024-01-15T14:35:06.420132+0000","last_clean":"2024-01-15T14:35:06.420132+0000","last_became_active":"2024-01-15T14:35:05.224306+0000","last_became_peered":"2024-01-15T14:35:05.224306+0000","last_unstale":"2024-01-15T14:35:06.420132+0000","last_undegraded":"2024-01-15T14:35:06.420132+0000","last_fullsized":"2024-01-15T14:35:06.420132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:35.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222848+0000","last_change":"2024-01-15T14:35:05.235613+0000","last_active":"2024-01-15T14:35:06.222848+0000","last_peered":"2024-01-15T14:35:06.222848+0000","last_clean":"2024-01-15T14:35:06.222848+0000","last_became_active":"2024-01-15T14:35:05.235298+0000","last_became_peered":"2024-01-15T14:35:05.235298+0000","last_unstale":"2024-01-15T14:35:06.222848+0000","last_undegraded":"2024-01-15T14:35:06.222848+0000","last_fullsized":"2024-01-15T14:35:06.222848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914462+0000","last_change":"2024-01-15T14:35:04.217677+0000","last_active":"2024-01-15T14:35:06.914462+0000","last_peered":"2024-01-15T14:35:06.914462+0000","last_clean":"2024-01-15T14:35:06.914462+0000","last_became_active":"2024-01-15T14:35:04.217361+0000","last_became_peered":"2024-01-15T14:35:04.217361+0000","last_unstale":"2024-01-15T14:35:06.914462+0000","last_undegraded":"2024-01-15T14:35:06.914462+0000","last_fullsized":"2024-01-15T14:35:06.914462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220970+0000","last_change":"2024-01-15T14:35:05.225192+0000","last_active":"2024-01-15T14:35:06.220970+0000","last_peered":"2024-01-15T14:35:06.220970+0000","last_clean":"2024-01-15T14:35:06.220970+0000","last_became_active":"2024-01-15T14:35:05.224634+0000","last_became_peered":"2024-01-15T14:35:05.224634+0000","last_unstale":"2024-01-15T14:35:06.220970+0000","last_undegraded":"2024-01-15T14:35:06.220970+0000","last_fullsized":"2024-01-15T14:35:06.220970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220874+0000","last_change":"2024-01-15T14:35:04.204838+0000","last_active":"2024-01-15T14:35:06.220874+0000","last_peered":"2024-01-15T14:35:06.220874+0000","last_clean":"2024-01-15T14:35:06.220874+0000","last_became_active":"2024-01-15T14:35:04.203651+0000","last_became_peered":"2024-01-15T14:35:04.203651+0000","last_unstale":"2024-01-15T14:35:06.220874+0000","last_undegraded":"2024-01-15T14:35:06.220874+0000","last_fullsized":"2024-01-15T14:35:06.220874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914437+0000","last_change":"2024-01-15T14:35:05.228513+0000","last_active":"2024-01-15T14:35:06.914437+0000","last_peered":"2024-01-15T14:35:06.914437+0000","last_clean":"2024-01-15T14:35:06.914437+0000","last_became_active":"2024-01-15T14:35:05.228229+0000","last_became_peered":"2024-01-15T14:35:05.228229+0000","last_unstale":"2024-01-15T14:35:06.914437+0000","last_undegraded":"2024-01-15T14:35:06.914437+0000","last_fullsized":"2024-01-15T14:35:06.914437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223203+0000","last_change":"2024-01-15T14:35:04.217602+0000","last_active":"2024-01-15T14:35:06.223203+0000","last_peered":"2024-01-15T14:35:06.223203+0000","last_clean":"2024-01-15T14:35:06.223203+0000","last_became_active":"2024-01-15T14:35:04.217271+0000","last_became_peered":"2024-01-15T14:35:04.217271+0000","last_unstale":"2024-01-15T14:35:06.223203+0000","last_undegraded":"2024-01-15T14:35:06.223203+0000","last_fullsized":"2024-01-15T14:35:06.223203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:24.748909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337912+0000","last_change":"2024-01-15T14:35:05.234884+0000","last_active":"2024-01-15T14:35:06.337912+0000","last_peered":"2024-01-15T14:35:06.337912+0000","last_clean":"2024-01-15T14:35:06.337912+0000","last_became_active":"2024-01-15T14:35:05.234556+0000","last_became_peered":"2024-01-15T14:35:05.234556+0000","last_unstale":"2024-01-15T14:35:06.337912+0000","last_undegraded":"2024-01-15T14:35:06.337912+0000","last_fullsized":"2024-01-15T14:35:06.337912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222800+0000","last_change":"2024-01-15T14:35:04.217549+0000","last_active":"2024-01-15T14:35:06.222800+0000","last_peered":"2024-01-15T14:35:06.222800+0000","last_clean":"2024-01-15T14:35:06.222800+0000","last_became_active":"2024-01-15T14:35:04.217245+0000","last_became_peered":"2024-01-15T14:35:04.217245+0000","last_unstale":"2024-01-15T14:35:06.222800+0000","last_undegraded":"2024-01-15T14:35:06.222800+0000","last_fullsized":"2024-01-15T14:35:06.222800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337934+0000","last_change":"2024-01-15T14:35:05.230835+0000","last_active":"2024-01-15T14:35:06.337934+0000","last_peered":"2024-01-15T14:35:06.337934+0000","last_clean":"2024-01-15T14:35:06.337934+0000","last_became_active":"2024-01-15T14:35:05.230395+0000","last_became_peered":"2024-01-15T14:35:05.230395+0000","last_unstale":"2024-01-15T14:35:06.337934+0000","last_undegraded":"2024-01-15T14:35:06.337934+0000","last_fullsized":"2024-01-15T14:35:06.337934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220762+0000","last_change":"2024-01-15T14:35:04.216108+0000","last_active":"2024-01-15T14:35:06.220762+0000","last_peered":"2024-01-15T14:35:06.220762+0000","last_clean":"2024-01-15T14:35:06.220762+0000","last_became_active":"2024-01-15T14:35:04.215774+0000","last_became_peered":"2024-01-15T14:35:04.215774+0000","last_unstale":"2024-01-15T14:35:06.220762+0000","last_undegraded":"2024-01-15T14:35:06.220762+0000","last_fullsized":"2024-01-15T14:35:06.220762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038274+0000","last_change":"2024-01-15T14:35:05.243121+0000","last_active":"2024-01-15T14:35:07.038274+0000","last_peered":"2024-01-15T14:35:07.038274+0000","last_clean":"2024-01-15T14:35:07.038274+0000","last_became_active":"2024-01-15T14:35:05.242858+0000","last_became_peered":"2024-01-15T14:35:05.242858+0000","last_unstale":"2024-01-15T14:35:07.038274+0000","last_undegraded":"2024-01-15T14:35:07.038274+0000","last_fullsized":"2024-01-15T14:35:07.038274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:38.527435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222109+0000","last_change":"2024-01-15T14:35:04.214194+0000","last_active":"2024-01-15T14:35:06.222109+0000","last_peered":"2024-01-15T14:35:06.222109+0000","last_clean":"2024-01-15T14:35:06.222109+0000","last_became_active":"2024-01-15T14:35:04.213952+0000","last_became_peered":"2024-01-15T14:35:04.213952+0000","last_unstale":"2024-01-15T14:35:06.222109+0000","last_undegraded":"2024-01-15T14:35:06.222109+0000","last_fullsized":"2024-01-15T14:35:06.222109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914552+0000","last_change":"2024-01-15T14:35:05.237633+0000","last_active":"2024-01-15T14:35:06.914552+0000","last_peered":"2024-01-15T14:35:06.914552+0000","last_clean":"2024-01-15T14:35:06.914552+0000","last_became_active":"2024-01-15T14:35:05.234559+0000","last_became_peered":"2024-01-15T14:35:05.234559+0000","last_unstale":"2024-01-15T14:35:06.914552+0000","last_undegraded":"2024-01-15T14:35:06.914552+0000","last_fullsized":"2024-01-15T14:35:06.914552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220477+0000","last_change":"2024-01-15T14:35:04.212481+0000","last_active":"2024-01-15T14:35:06.220477+0000","last_peered":"2024-01-15T14:35:06.220477+0000","last_clean":"2024-01-15T14:35:06.220477+0000","last_became_active":"2024-01-15T14:35:04.212205+0000","last_became_peered":"2024-01-15T14:35:04.212205+0000","last_unstale":"2024-01-15T14:35:06.220477+0000","last_undegraded":"2024-01-15T14:35:06.220477+0000","last_fullsized":"2024-01-15T14:35:06.220477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220657+0000","last_change":"2024-01-15T14:35:05.229905+0000","last_active":"2024-01-15T14:35:06.220657+0000","last_peered":"2024-01-15T14:35:06.220657+0000","last_clean":"2024-01-15T14:35:06.220657+0000","last_became_active":"2024-01-15T14:35:05.229591+0000","last_became_peered":"2024-01-15T14:35:05.229591+0000","last_unstale":"2024-01-15T14:35:06.220657+0000","last_undegraded":"2024-01-15T14:35:06.220657+0000","last_fullsized":"2024-01-15T14:35:06.220657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337590+0000","last_change":"2024-01-15T14:35:04.212238+0000","last_active":"2024-01-15T14:35:06.337590+0000","last_peered":"2024-01-15T14:35:06.337590+0000","last_clean":"2024-01-15T14:35:06.337590+0000","last_became_active":"2024-01-15T14:35:04.211931+0000","last_became_peered":"2024-01-15T14:35:04.211931+0000","last_unstale":"2024-01-15T14:35:06.337590+0000","last_undegraded":"2024-01-15T14:35:06.337590+0000","last_fullsized":"2024-01-15T14:35:06.337590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:39.371648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038653+0000","last_change":"2024-01-15T14:35:05.215883+0000","last_active":"2024-01-15T14:35:07.038653+0000","last_peered":"2024-01-15T14:35:07.038653+0000","last_clean":"2024-01-15T14:35:07.038653+0000","last_became_active":"2024-01-15T14:35:05.215584+0000","last_became_peered":"2024-01-15T14:35:05.215584+0000","last_unstale":"2024-01-15T14:35:07.038653+0000","last_undegraded":"2024-01-15T14:35:07.038653+0000","last_fullsized":"2024-01-15T14:35:07.038653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222538+0000","last_change":"2024-01-15T14:35:04.212959+0000","last_active":"2024-01-15T14:35:06.222538+0000","last_peered":"2024-01-15T14:35:06.222538+0000","last_clean":"2024-01-15T14:35:06.222538+0000","last_became_active":"2024-01-15T14:35:04.212632+0000","last_became_peered":"2024-01-15T14:35:04.212632+0000","last_unstale":"2024-01-15T14:35:06.222538+0000","last_undegraded":"2024-01-15T14:35:06.222538+0000","last_fullsized":"2024-01-15T14:35:06.222538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337983+0000","last_change":"2024-01-15T14:35:05.232646+0000","last_active":"2024-01-15T14:35:06.337983+0000","last_peered":"2024-01-15T14:35:06.337983+0000","last_clean":"2024-01-15T14:35:06.337983+0000","last_became_active":"2024-01-15T14:35:05.232360+0000","last_became_peered":"2024-01-15T14:35:05.232360+0000","last_unstale":"2024-01-15T14:35:06.337983+0000","last_undegraded":"2024-01-15T14:35:06.337983+0000","last_fullsized":"2024-01-15T14:35:06.337983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337788+0000","last_change":"2024-01-15T14:35:04.207794+0000","last_active":"2024-01-15T14:35:06.337788+0000","last_peered":"2024-01-15T14:35:06.337788+0000","last_clean":"2024-01-15T14:35:06.337788+0000","last_became_active":"2024-01-15T14:35:04.207503+0000","last_became_peered":"2024-01-15T14:35:04.207503+0000","last_unstale":"2024-01-15T14:35:06.337788+0000","last_undegraded":"2024-01-15T14:35:06.337788+0000","last_fullsized":"2024-01-15T14:35:06.337788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914599+0000","last_change":"2024-01-15T14:35:05.228000+0000","last_active":"2024-01-15T14:35:06.914599+0000","last_peered":"2024-01-15T14:35:06.914599+0000","last_clean":"2024-01-15T14:35:06.914599+0000","last_became_active":"2024-01-15T14:35:05.227713+0000","last_became_peered":"2024-01-15T14:35:05.227713+0000","last_unstale":"2024-01-15T14:35:06.914599+0000","last_undegraded":"2024-01-15T14:35:06.914599+0000","last_fullsized":"2024-01-15T14:35:06.914599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038760+0000","last_change":"2024-01-15T14:35:04.212694+0000","last_active":"2024-01-15T14:35:07.038760+0000","last_peered":"2024-01-15T14:35:07.038760+0000","last_clean":"2024-01-15T14:35:07.038760+0000","last_became_active":"2024-01-15T14:35:04.212368+0000","last_became_peered":"2024-01-15T14:35:04.212368+0000","last_unstale":"2024-01-15T14:35:07.038760+0000","last_undegraded":"2024-01-15T14:35:07.038760+0000","last_fullsized":"2024-01-15T14:35:07.038760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038591+0000","last_change":"2024-01-15T14:35:05.235186+0000","last_active":"2024-01-15T14:35:07.038591+0000","last_peered":"2024-01-15T14:35:07.038591+0000","last_clean":"2024-01-15T14:35:07.038591+0000","last_became_active":"2024-01-15T14:35:05.234887+0000","last_became_peered":"2024-01-15T14:35:05.234887+0000","last_unstale":"2024-01-15T14:35:07.038591+0000","last_undegraded":"2024-01-15T14:35:07.038591+0000","last_fullsized":"2024-01-15T14:35:07.038591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:43.060852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222002+0000","last_change":"2024-01-15T14:35:04.212306+0000","last_active":"2024-01-15T14:35:06.222002+0000","last_peered":"2024-01-15T14:35:06.222002+0000","last_clean":"2024-01-15T14:35:06.222002+0000","last_became_active":"2024-01-15T14:35:04.211871+0000","last_became_peered":"2024-01-15T14:35:04.211871+0000","last_unstale":"2024-01-15T14:35:06.222002+0000","last_undegraded":"2024-01-15T14:35:06.222002+0000","last_fullsized":"2024-01-15T14:35:06.222002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223019+0000","last_change":"2024-01-15T14:35:05.242305+0000","last_active":"2024-01-15T14:35:06.223019+0000","last_peered":"2024-01-15T14:35:06.223019+0000","last_clean":"2024-01-15T14:35:06.223019+0000","last_became_active":"2024-01-15T14:35:05.242026+0000","last_became_peered":"2024-01-15T14:35:05.242026+0000","last_unstale":"2024-01-15T14:35:06.223019+0000","last_undegraded":"2024-01-15T14:35:06.223019+0000","last_fullsized":"2024-01-15T14:35:06.223019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914403+0000","last_change":"2024-01-15T14:35:04.205419+0000","last_active":"2024-01-15T14:35:06.914403+0000","last_peered":"2024-01-15T14:35:06.914403+0000","last_clean":"2024-01-15T14:35:06.914403+0000","last_became_active":"2024-01-15T14:35:04.205106+0000","last_became_peered":"2024-01-15T14:35:04.205106+0000","last_unstale":"2024-01-15T14:35:06.914403+0000","last_undegraded":"2024-01-15T14:35:06.914403+0000","last_fullsized":"2024-01-15T14:35:06.914403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914366+0000","last_change":"2024-01-15T14:35:05.239426+0000","last_active":"2024-01-15T14:35:06.914366+0000","last_peered":"2024-01-15T14:35:06.914366+0000","last_clean":"2024-01-15T14:35:06.914366+0000","last_became_active":"2024-01-15T14:35:05.239139+0000","last_became_peered":"2024-01-15T14:35:05.239139+0000","last_unstale":"2024-01-15T14:35:06.914366+0000","last_undegraded":"2024-01-15T14:35:06.914366+0000","last_fullsized":"2024-01-15T14:35:06.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338008+0000","last_change":"2024-01-15T14:35:04.207140+0000","last_active":"2024-01-15T14:35:06.338008+0000","last_peered":"2024-01-15T14:35:06.338008+0000","last_clean":"2024-01-15T14:35:06.338008+0000","last_became_active":"2024-01-15T14:35:04.206804+0000","last_became_peered":"2024-01-15T14:35:04.206804+0000","last_unstale":"2024-01-15T14:35:06.338008+0000","last_undegraded":"2024-01-15T14:35:06.338008+0000","last_fullsized":"2024-01-15T14:35:06.338008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:30.291286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222686+0000","last_change":"2024-01-15T14:35:05.240267+0000","last_active":"2024-01-15T14:35:06.222686+0000","last_peered":"2024-01-15T14:35:06.222686+0000","last_clean":"2024-01-15T14:35:06.222686+0000","last_became_active":"2024-01-15T14:35:05.239993+0000","last_became_peered":"2024-01-15T14:35:05.239993+0000","last_unstale":"2024-01-15T14:35:06.222686+0000","last_undegraded":"2024-01-15T14:35:06.222686+0000","last_fullsized":"2024-01-15T14:35:06.222686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221632+0000","last_change":"2024-01-15T14:35:04.214296+0000","last_active":"2024-01-15T14:35:06.221632+0000","last_peered":"2024-01-15T14:35:06.221632+0000","last_clean":"2024-01-15T14:35:06.221632+0000","last_became_active":"2024-01-15T14:35:04.213992+0000","last_became_peered":"2024-01-15T14:35:04.213992+0000","last_unstale":"2024-01-15T14:35:06.221632+0000","last_undegraded":"2024-01-15T14:35:06.221632+0000","last_fullsized":"2024-01-15T14:35:06.221632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221677+0000","last_change":"2024-01-15T14:35:05.231106+0000","last_active":"2024-01-15T14:35:06.221677+0000","last_peered":"2024-01-15T14:35:06.221677+0000","last_clean":"2024-01-15T14:35:06.221677+0000","last_became_active":"2024-01-15T14:35:05.230594+0000","last_became_peered":"2024-01-15T14:35:05.230594+0000","last_unstale":"2024-01-15T14:35:06.221677+0000","last_undegraded":"2024-01-15T14:35:06.221677+0000","last_fullsized":"2024-01-15T14:35:06.221677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420056+0000","last_change":"2024-01-15T14:35:04.211246+0000","last_active":"2024-01-15T14:35:06.420056+0000","last_peered":"2024-01-15T14:35:06.420056+0000","last_clean":"2024-01-15T14:35:06.420056+0000","last_became_active":"2024-01-15T14:35:04.210971+0000","last_became_peered":"2024-01-15T14:35:04.210971+0000","last_unstale":"2024-01-15T14:35:06.420056+0000","last_undegraded":"2024-01-15T14:35:06.420056+0000","last_fullsized":"2024-01-15T14:35:06.420056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:06.847713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222962+0000","last_change":"2024-01-15T14:35:05.240373+0000","last_active":"2024-01-15T14:35:06.222962+0000","last_peered":"2024-01-15T14:35:06.222962+0000","last_clean":"2024-01-15T14:35:06.222962+0000","last_became_active":"2024-01-15T14:35:05.240078+0000","last_became_peered":"2024-01-15T14:35:05.240078+0000","last_unstale":"2024-01-15T14:35:06.222962+0000","last_undegraded":"2024-01-15T14:35:06.222962+0000","last_fullsized":"2024-01-15T14:35:06.222962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038437+0000","last_change":"2024-01-15T14:35:04.204121+0000","last_active":"2024-01-15T14:35:07.038437+0000","last_peered":"2024-01-15T14:35:07.038437+0000","last_clean":"2024-01-15T14:35:07.038437+0000","last_became_active":"2024-01-15T14:35:04.203811+0000","last_became_peered":"2024-01-15T14:35:04.203811+0000","last_unstale":"2024-01-15T14:35:07.038437+0000","last_undegraded":"2024-01-15T14:35:07.038437+0000","last_fullsized":"2024-01-15T14:35:07.038437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222934+0000","last_change":"2024-01-15T14:35:05.239960+0000","last_active":"2024-01-15T14:35:06.222934+0000","last_peered":"2024-01-15T14:35:06.222934+0000","last_clean":"2024-01-15T14:35:06.222934+0000","last_became_active":"2024-01-15T14:35:05.239645+0000","last_became_peered":"2024-01-15T14:35:05.239645+0000","last_unstale":"2024-01-15T14:35:06.222934+0000","last_undegraded":"2024-01-15T14:35:06.222934+0000","last_fullsized":"2024-01-15T14:35:06.222934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038563+0000","last_change":"2024-01-15T14:35:04.212055+0000","last_active":"2024-01-15T14:35:07.038563+0000","last_peered":"2024-01-15T14:35:07.038563+0000","last_clean":"2024-01-15T14:35:07.038563+0000","last_became_active":"2024-01-15T14:35:04.211788+0000","last_became_peered":"2024-01-15T14:35:04.211788+0000","last_unstale":"2024-01-15T14:35:07.038563+0000","last_undegraded":"2024-01-15T14:35:07.038563+0000","last_fullsized":"2024-01-15T14:35:07.038563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338047+0000","last_change":"2024-01-15T14:35:05.220108+0000","last_active":"2024-01-15T14:35:06.338047+0000","last_peered":"2024-01-15T14:35:06.338047+0000","last_clean":"2024-01-15T14:35:06.338047+0000","last_became_active":"2024-01-15T14:35:05.219812+0000","last_became_peered":"2024-01-15T14:35:05.219812+0000","last_unstale":"2024-01-15T14:35:06.338047+0000","last_undegraded":"2024-01-15T14:35:06.338047+0000","last_fullsized":"2024-01-15T14:35:06.338047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337937+0000","last_change":"2024-01-15T14:35:04.212432+0000","last_active":"2024-01-15T14:35:06.337937+0000","last_peered":"2024-01-15T14:35:06.337937+0000","last_clean":"2024-01-15T14:35:06.337937+0000","last_became_active":"2024-01-15T14:35:04.211351+0000","last_became_peered":"2024-01-15T14:35:04.211351+0000","last_unstale":"2024-01-15T14:35:06.337937+0000","last_undegraded":"2024-01-15T14:35:06.337937+0000","last_fullsized":"2024-01-15T14:35:06.337937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:38.895254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038566+0000","last_change":"2024-01-15T14:35:05.234938+0000","last_active":"2024-01-15T14:35:07.038566+0000","last_peered":"2024-01-15T14:35:07.038566+0000","last_clean":"2024-01-15T14:35:07.038566+0000","last_became_active":"2024-01-15T14:35:05.234597+0000","last_became_peered":"2024-01-15T14:35:05.234597+0000","last_unstale":"2024-01-15T14:35:07.038566+0000","last_undegraded":"2024-01-15T14:35:07.038566+0000","last_fullsized":"2024-01-15T14:35:07.038566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:00.256458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038392+0000","last_change":"2024-01-15T14:35:05.211465+0000","last_active":"2024-01-15T14:35:07.038392+0000","last_peered":"2024-01-15T14:35:07.038392+0000","last_clean":"2024-01-15T14:35:07.038392+0000","last_became_active":"2024-01-15T14:35:05.211188+0000","last_became_peered":"2024-01-15T14:35:05.211188+0000","last_unstale":"2024-01-15T14:35:07.038392+0000","last_undegraded":"2024-01-15T14:35:07.038392+0000","last_fullsized":"2024-01-15T14:35:07.038392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.419968+0000","last_change":"2024-01-15T14:35:04.208525+0000","last_active":"2024-01-15T14:35:06.419968+0000","last_peered":"2024-01-15T14:35:06.419968+0000","last_clean":"2024-01-15T14:35:06.419968+0000","last_became_active":"2024-01-15T14:35:04.208217+0000","last_became_peered":"2024-01-15T14:35:04.208217+0000","last_unstale":"2024-01-15T14:35:06.419968+0000","last_undegraded":"2024-01-15T14:35:06.419968+0000","last_fullsized":"2024-01-15T14:35:06.419968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220595+0000","last_change":"2024-01-15T14:35:05.236274+0000","last_active":"2024-01-15T14:35:06.220595+0000","last_peered":"2024-01-15T14:35:06.220595+0000","last_clean":"2024-01-15T14:35:06.220595+0000","last_became_active":"2024-01-15T14:35:05.236000+0000","last_became_peered":"2024-01-15T14:35:05.236000+0000","last_unstale":"2024-01-15T14:35:06.220595+0000","last_undegraded":"2024-01-15T14:35:06.220595+0000","last_fullsized":"2024-01-15T14:35:06.220595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337639+0000","last_change":"2024-01-15T14:35:04.209747+0000","last_active":"2024-01-15T14:35:06.337639+0000","last_peered":"2024-01-15T14:35:06.337639+0000","last_clean":"2024-01-15T14:35:06.337639+0000","last_became_active":"2024-01-15T14:35:04.209450+0000","last_became_peered":"2024-01-15T14:35:04.209450+0000","last_unstale":"2024-01-15T14:35:06.337639+0000","last_undegraded":"2024-01-15T14:35:06.337639+0000","last_fullsized":"2024-01-15T14:35:06.337639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:37:47.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T14:35:05.216923+0000","last_change":"2024-01-15T14:34:54.738502+0000","last_active":"2024-01-15T14:35:05.216923+0000","last_peered":"2024-01-15T14:35:05.216923+0000","last_clean":"2024-01-15T14:35:05.216923+0000","last_became_active":"2024-01-15T14:34:54.737354+0000","last_became_peered":"2024-01-15T14:34:54.737354+0000","last_unstale":"2024-01-15T14:35:05.216923+0000","last_undegraded":"2024-01-15T14:35:05.216923+0000","last_fullsized":"2024-01-15T14:35:05.216923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_clean_scrub_stamp":"2024-01-15T14:34:53.124419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:09:14.038173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914075+0000","last_change":"2024-01-15T14:35:05.231021+0000","last_active":"2024-01-15T14:35:06.914075+0000","last_peered":"2024-01-15T14:35:06.914075+0000","last_clean":"2024-01-15T14:35:06.914075+0000","last_became_active":"2024-01-15T14:35:05.230721+0000","last_became_peered":"2024-01-15T14:35:05.230721+0000","last_unstale":"2024-01-15T14:35:06.914075+0000","last_undegraded":"2024-01-15T14:35:06.914075+0000","last_fullsized":"2024-01-15T14:35:06.914075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:15.565484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222435+0000","last_change":"2024-01-15T14:35:04.216271+0000","last_active":"2024-01-15T14:35:06.222435+0000","last_peered":"2024-01-15T14:35:06.222435+0000","last_clean":"2024-01-15T14:35:06.222435+0000","last_became_active":"2024-01-15T14:35:04.215955+0000","last_became_peered":"2024-01-15T14:35:04.215955+0000","last_unstale":"2024-01-15T14:35:06.222435+0000","last_undegraded":"2024-01-15T14:35:06.222435+0000","last_fullsized":"2024-01-15T14:35:06.222435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223126+0000","last_change":"2024-01-15T14:35:04.218008+0000","last_active":"2024-01-15T14:35:06.223126+0000","last_peered":"2024-01-15T14:35:06.223126+0000","last_clean":"2024-01-15T14:35:06.223126+0000","last_became_active":"2024-01-15T14:35:04.217672+0000","last_became_peered":"2024-01-15T14:35:04.217672+0000","last_unstale":"2024-01-15T14:35:06.223126+0000","last_undegraded":"2024-01-15T14:35:06.223126+0000","last_fullsized":"2024-01-15T14:35:06.223126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:25.883807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038622+0000","last_change":"2024-01-15T14:35:05.213577+0000","last_active":"2024-01-15T14:35:07.038622+0000","last_peered":"2024-01-15T14:35:07.038622+0000","last_clean":"2024-01-15T14:35:07.038622+0000","last_became_active":"2024-01-15T14:35:05.212565+0000","last_became_peered":"2024-01-15T14:35:05.212565+0000","last_unstale":"2024-01-15T14:35:07.038622+0000","last_undegraded":"2024-01-15T14:35:07.038622+0000","last_fullsized":"2024-01-15T14:35:07.038622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.106745+0000","last_change":"2024-01-15T14:35:04.216515+0000","last_active":"2024-01-15T14:35:07.106745+0000","last_peered":"2024-01-15T14:35:07.106745+0000","last_clean":"2024-01-15T14:35:07.106745+0000","last_became_active":"2024-01-15T14:35:04.216291+0000","last_became_peered":"2024-01-15T14:35:04.216291+0000","last_unstale":"2024-01-15T14:35:07.106745+0000","last_undegraded":"2024-01-15T14:35:07.106745+0000","last_fullsized":"2024-01-15T14:35:07.106745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221002+0000","last_change":"2024-01-15T14:35:05.233144+0000","last_active":"2024-01-15T14:35:06.221002+0000","last_peered":"2024-01-15T14:35:06.221002+0000","last_clean":"2024-01-15T14:35:06.221002+0000","last_became_active":"2024-01-15T14:35:05.232813+0000","last_became_peered":"2024-01-15T14:35:05.232813+0000","last_unstale":"2024-01-15T14:35:06.221002+0000","last_undegraded":"2024-01-15T14:35:06.221002+0000","last_fullsized":"2024-01-15T14:35:06.221002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222447+0000","last_change":"2024-01-15T14:35:04.215265+0000","last_active":"2024-01-15T14:35:06.222447+0000","last_peered":"2024-01-15T14:35:06.222447+0000","last_clean":"2024-01-15T14:35:06.222447+0000","last_became_active":"2024-01-15T14:35:04.214951+0000","last_became_peered":"2024-01-15T14:35:04.214951+0000","last_unstale":"2024-01-15T14:35:06.222447+0000","last_undegraded":"2024-01-15T14:35:06.222447+0000","last_fullsized":"2024-01-15T14:35:06.222447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914377+0000","last_change":"2024-01-15T14:35:05.228381+0000","last_active":"2024-01-15T14:35:06.914377+0000","last_peered":"2024-01-15T14:35:06.914377+0000","last_clean":"2024-01-15T14:35:06.914377+0000","last_became_active":"2024-01-15T14:35:05.228101+0000","last_became_peered":"2024-01-15T14:35:05.228101+0000","last_unstale":"2024-01-15T14:35:06.914377+0000","last_undegraded":"2024-01-15T14:35:06.914377+0000","last_fullsized":"2024-01-15T14:35:06.914377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222345+0000","last_change":"2024-01-15T14:35:04.215917+0000","last_active":"2024-01-15T14:35:06.222345+0000","last_peered":"2024-01-15T14:35:06.222345+0000","last_clean":"2024-01-15T14:35:06.222345+0000","last_became_active":"2024-01-15T14:35:04.215592+0000","last_became_peered":"2024-01-15T14:35:04.215592+0000","last_unstale":"2024-01-15T14:35:06.222345+0000","last_undegraded":"2024-01-15T14:35:06.222345+0000","last_fullsized":"2024-01-15T14:35:06.222345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038319+0000","last_change":"2024-01-15T14:35:05.233251+0000","last_active":"2024-01-15T14:35:07.038319+0000","last_peered":"2024-01-15T14:35:07.038319+0000","last_clean":"2024-01-15T14:35:07.038319+0000","last_became_active":"2024-01-15T14:35:05.232982+0000","last_became_peered":"2024-01-15T14:35:05.232982+0000","last_unstale":"2024-01-15T14:35:07.038319+0000","last_undegraded":"2024-01-15T14:35:07.038319+0000","last_fullsized":"2024-01-15T14:35:07.038319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:15.827108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038435+0000","last_change":"2024-01-15T14:35:04.218539+0000","last_active":"2024-01-15T14:35:07.038435+0000","last_peered":"2024-01-15T14:35:07.038435+0000","last_clean":"2024-01-15T14:35:07.038435+0000","last_became_active":"2024-01-15T14:35:04.218307+0000","last_became_peered":"2024-01-15T14:35:04.218307+0000","last_unstale":"2024-01-15T14:35:07.038435+0000","last_undegraded":"2024-01-15T14:35:07.038435+0000","last_fullsized":"2024-01-15T14:35:07.038435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220625+0000","last_change":"2024-01-15T14:35:05.220403+0000","last_active":"2024-01-15T14:35:06.220625+0000","last_peered":"2024-01-15T14:35:06.220625+0000","last_clean":"2024-01-15T14:35:06.220625+0000","last_became_active":"2024-01-15T14:35:05.220033+0000","last_became_peered":"2024-01-15T14:35:05.220033+0000","last_unstale":"2024-01-15T14:35:06.220625+0000","last_undegraded":"2024-01-15T14:35:06.220625+0000","last_fullsized":"2024-01-15T14:35:06.220625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038582+0000","last_change":"2024-01-15T14:35:04.211061+0000","last_active":"2024-01-15T14:35:07.038582+0000","last_peered":"2024-01-15T14:35:07.038582+0000","last_clean":"2024-01-15T14:35:07.038582+0000","last_became_active":"2024-01-15T14:35:04.210748+0000","last_became_peered":"2024-01-15T14:35:04.210748+0000","last_unstale":"2024-01-15T14:35:07.038582+0000","last_undegraded":"2024-01-15T14:35:07.038582+0000","last_fullsized":"2024-01-15T14:35:07.038582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220819+0000","last_change":"2024-01-15T14:35:05.232247+0000","last_active":"2024-01-15T14:35:06.220819+0000","last_peered":"2024-01-15T14:35:06.220819+0000","last_clean":"2024-01-15T14:35:06.220819+0000","last_became_active":"2024-01-15T14:35:05.231914+0000","last_became_peered":"2024-01-15T14:35:05.231914+0000","last_unstale":"2024-01-15T14:35:06.220819+0000","last_undegraded":"2024-01-15T14:35:06.220819+0000","last_fullsized":"2024-01-15T14:35:06.220819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222720+0000","last_change":"2024-01-15T14:35:04.216586+0000","last_active":"2024-01-15T14:35:06.222720+0000","last_peered":"2024-01-15T14:35:06.222720+0000","last_clean":"2024-01-15T14:35:06.222720+0000","last_became_active":"2024-01-15T14:35:04.216231+0000","last_became_peered":"2024-01-15T14:35:04.216231+0000","last_unstale":"2024-01-15T14:35:06.222720+0000","last_undegraded":"2024-01-15T14:35:06.222720+0000","last_fullsized":"2024-01-15T14:35:06.222720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221812+0000","last_change":"2024-01-15T14:35:05.237033+0000","last_active":"2024-01-15T14:35:06.221812+0000","last_peered":"2024-01-15T14:35:06.221812+0000","last_clean":"2024-01-15T14:35:06.221812+0000","last_became_active":"2024-01-15T14:35:05.236738+0000","last_became_peered":"2024-01-15T14:35:05.236738+0000","last_unstale":"2024-01-15T14:35:06.221812+0000","last_undegraded":"2024-01-15T14:35:06.221812+0000","last_fullsized":"2024-01-15T14:35:06.221812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420017+0000","last_change":"2024-01-15T14:35:04.208500+0000","last_active":"2024-01-15T14:35:06.420017+0000","last_peered":"2024-01-15T14:35:06.420017+0000","last_clean":"2024-01-15T14:35:06.420017+0000","last_became_active":"2024-01-15T14:35:04.208227+0000","last_became_peered":"2024-01-15T14:35:04.208227+0000","last_unstale":"2024-01-15T14:35:06.420017+0000","last_undegraded":"2024-01-15T14:35:06.420017+0000","last_fullsized":"2024-01-15T14:35:06.420017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:11:11.486899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223119+0000","last_change":"2024-01-15T14:35:05.232373+0000","last_active":"2024-01-15T14:35:06.223119+0000","last_peered":"2024-01-15T14:35:06.223119+0000","last_clean":"2024-01-15T14:35:06.223119+0000","last_became_active":"2024-01-15T14:35:05.232098+0000","last_became_peered":"2024-01-15T14:35:05.232098+0000","last_unstale":"2024-01-15T14:35:06.223119+0000","last_undegraded":"2024-01-15T14:35:06.223119+0000","last_fullsized":"2024-01-15T14:35:06.223119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038754+0000","last_change":"2024-01-15T14:35:04.213296+0000","last_active":"2024-01-15T14:35:07.038754+0000","last_peered":"2024-01-15T14:35:07.038754+0000","last_clean":"2024-01-15T14:35:07.038754+0000","last_became_active":"2024-01-15T14:35:04.212955+0000","last_became_peered":"2024-01-15T14:35:04.212955+0000","last_unstale":"2024-01-15T14:35:07.038754+0000","last_undegraded":"2024-01-15T14:35:07.038754+0000","last_fullsized":"2024-01-15T14:35:07.038754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:10.265464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338036+0000","last_change":"2024-01-15T14:35:05.219979+0000","last_active":"2024-01-15T14:35:06.338036+0000","last_peered":"2024-01-15T14:35:06.338036+0000","last_clean":"2024-01-15T14:35:06.338036+0000","last_became_active":"2024-01-15T14:35:05.219668+0000","last_became_peered":"2024-01-15T14:35:05.219668+0000","last_unstale":"2024-01-15T14:35:06.338036+0000","last_undegraded":"2024-01-15T14:35:06.338036+0000","last_fullsized":"2024-01-15T14:35:06.338036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420118+0000","last_change":"2024-01-15T14:35:04.208970+0000","last_active":"2024-01-15T14:35:06.420118+0000","last_peered":"2024-01-15T14:35:06.420118+0000","last_clean":"2024-01-15T14:35:06.420118+0000","last_became_active":"2024-01-15T14:35:04.208699+0000","last_became_peered":"2024-01-15T14:35:04.208699+0000","last_unstale":"2024-01-15T14:35:06.420118+0000","last_undegraded":"2024-01-15T14:35:06.420118+0000","last_fullsized":"2024-01-15T14:35:06.420118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223420+0000","last_change":"2024-01-15T14:35:05.239873+0000","last_active":"2024-01-15T14:35:06.223420+0000","last_peered":"2024-01-15T14:35:06.223420+0000","last_clean":"2024-01-15T14:35:06.223420+0000","last_became_active":"2024-01-15T14:35:05.239571+0000","last_became_peered":"2024-01-15T14:35:05.239571+0000","last_unstale":"2024-01-15T14:35:06.223420+0000","last_undegraded":"2024-01-15T14:35:06.223420+0000","last_fullsized":"2024-01-15T14:35:06.223420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221810+0000","last_change":"2024-01-15T14:35:05.230887+0000","last_active":"2024-01-15T14:35:06.221810+0000","last_peered":"2024-01-15T14:35:06.221810+0000","last_clean":"2024-01-15T14:35:06.221810+0000","last_became_active":"2024-01-15T14:35:05.230187+0000","last_became_peered":"2024-01-15T14:35:05.230187+0000","last_unstale":"2024-01-15T14:35:06.221810+0000","last_undegraded":"2024-01-15T14:35:06.221810+0000","last_fullsized":"2024-01-15T14:35:06.221810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420110+0000","last_change":"2024-01-15T14:35:05.217285+0000","last_active":"2024-01-15T14:35:06.420110+0000","last_peered":"2024-01-15T14:35:06.420110+0000","last_clean":"2024-01-15T14:35:06.420110+0000","last_became_active":"2024-01-15T14:35:05.217015+0000","last_became_peered":"2024-01-15T14:35:05.217015+0000","last_unstale":"2024-01-15T14:35:06.420110+0000","last_undegraded":"2024-01-15T14:35:06.420110+0000","last_fullsized":"2024-01-15T14:35:06.420110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337918+0000","last_change":"2024-01-15T14:35:05.220986+0000","last_active":"2024-01-15T14:35:06.337918+0000","last_peered":"2024-01-15T14:35:06.337918+0000","last_clean":"2024-01-15T14:35:06.337918+0000","last_became_active":"2024-01-15T14:35:05.220704+0000","last_became_peered":"2024-01-15T14:35:05.220704+0000","last_unstale":"2024-01-15T14:35:06.337918+0000","last_undegraded":"2024-01-15T14:35:06.337918+0000","last_fullsized":"2024-01-15T14:35:06.337918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914367+0000","last_change":"2024-01-15T14:35:05.232048+0000","last_active":"2024-01-15T14:35:06.914367+0000","last_peered":"2024-01-15T14:35:06.914367+0000","last_clean":"2024-01-15T14:35:06.914367+0000","last_became_active":"2024-01-15T14:35:05.231708+0000","last_became_peered":"2024-01-15T14:35:05.231708+0000","last_unstale":"2024-01-15T14:35:06.914367+0000","last_undegraded":"2024-01-15T14:35:06.914367+0000","last_fullsized":"2024-01-15T14:35:06.914367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220904+0000","last_change":"2024-01-15T14:35:05.236304+0000","last_active":"2024-01-15T14:35:06.220904+0000","last_peered":"2024-01-15T14:35:06.220904+0000","last_clean":"2024-01-15T14:35:06.220904+0000","last_became_active":"2024-01-15T14:35:05.235435+0000","last_became_peered":"2024-01-15T14:35:05.235435+0000","last_unstale":"2024-01-15T14:35:06.220904+0000","last_undegraded":"2024-01-15T14:35:06.220904+0000","last_fullsized":"2024-01-15T14:35:06.220904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038669+0000","last_change":"2024-01-15T14:35:05.216092+0000","last_active":"2024-01-15T14:35:07.038669+0000","last_peered":"2024-01-15T14:35:07.038669+0000","last_clean":"2024-01-15T14:35:07.038669+0000","last_became_active":"2024-01-15T14:35:05.215618+0000","last_became_peered":"2024-01-15T14:35:05.215618+0000","last_unstale":"2024-01-15T14:35:07.038669+0000","last_undegraded":"2024-01-15T14:35:07.038669+0000","last_fullsized":"2024-01-15T14:35:07.038669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222044+0000","last_change":"2024-01-15T14:35:05.229045+0000","last_active":"2024-01-15T14:35:06.222044+0000","last_peered":"2024-01-15T14:35:06.222044+0000","last_clean":"2024-01-15T14:35:06.222044+0000","last_became_active":"2024-01-15T14:35:05.228771+0000","last_became_peered":"2024-01-15T14:35:05.228771+0000","last_unstale":"2024-01-15T14:35:06.222044+0000","last_undegraded":"2024-01-15T14:35:06.222044+0000","last_fullsized":"2024-01-15T14:35:06.222044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220972+0000","last_change":"2024-01-15T14:35:05.235711+0000","last_active":"2024-01-15T14:35:06.220972+0000","last_peered":"2024-01-15T14:35:06.220972+0000","last_clean":"2024-01-15T14:35:06.220972+0000","last_became_active":"2024-01-15T14:35:05.235419+0000","last_became_peered":"2024-01-15T14:35:05.235419+0000","last_unstale":"2024-01-15T14:35:06.220972+0000","last_undegraded":"2024-01-15T14:35:06.220972+0000","last_fullsized":"2024-01-15T14:35:06.220972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337825+0000","last_change":"2024-01-15T14:35:05.221416+0000","last_active":"2024-01-15T14:35:06.337825+0000","last_peered":"2024-01-15T14:35:06.337825+0000","last_clean":"2024-01-15T14:35:06.337825+0000","last_became_active":"2024-01-15T14:35:05.221124+0000","last_became_peered":"2024-01-15T14:35:05.221124+0000","last_unstale":"2024-01-15T14:35:06.337825+0000","last_undegraded":"2024-01-15T14:35:06.337825+0000","last_fullsized":"2024-01-15T14:35:06.337825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220561+0000","last_change":"2024-01-15T14:35:05.235785+0000","last_active":"2024-01-15T14:35:06.220561+0000","last_peered":"2024-01-15T14:35:06.220561+0000","last_clean":"2024-01-15T14:35:06.220561+0000","last_became_active":"2024-01-15T14:35:05.235193+0000","last_became_peered":"2024-01-15T14:35:05.235193+0000","last_unstale":"2024-01-15T14:35:06.220561+0000","last_undegraded":"2024-01-15T14:35:06.220561+0000","last_fullsized":"2024-01-15T14:35:06.220561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337729+0000","last_change":"2024-01-15T14:35:05.225944+0000","last_active":"2024-01-15T14:35:06.337729+0000","last_peered":"2024-01-15T14:35:06.337729+0000","last_clean":"2024-01-15T14:35:06.337729+0000","last_became_active":"2024-01-15T14:35:05.225660+0000","last_became_peered":"2024-01-15T14:35:05.225660+0000","last_unstale":"2024-01-15T14:35:06.337729+0000","last_undegraded":"2024-01-15T14:35:06.337729+0000","last_fullsized":"2024-01-15T14:35:06.337729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914286+0000","last_change":"2024-01-15T14:35:05.243644+0000","last_active":"2024-01-15T14:35:06.914286+0000","last_peered":"2024-01-15T14:35:06.914286+0000","last_clean":"2024-01-15T14:35:06.914286+0000","last_became_active":"2024-01-15T14:35:05.243362+0000","last_became_peered":"2024-01-15T14:35:05.243362+0000","last_unstale":"2024-01-15T14:35:06.914286+0000","last_undegraded":"2024-01-15T14:35:06.914286+0000","last_fullsized":"2024-01-15T14:35:06.914286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:58.010415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222177+0000","last_change":"2024-01-15T14:35:05.237833+0000","last_active":"2024-01-15T14:35:06.222177+0000","last_peered":"2024-01-15T14:35:06.222177+0000","last_clean":"2024-01-15T14:35:06.222177+0000","last_became_active":"2024-01-15T14:35:05.237576+0000","last_became_peered":"2024-01-15T14:35:05.237576+0000","last_unstale":"2024-01-15T14:35:06.222177+0000","last_undegraded":"2024-01-15T14:35:06.222177+0000","last_fullsized":"2024-01-15T14:35:06.222177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9929999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0339999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.106000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.383}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.984999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0650000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4119999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T14:35:13.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T14:35:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.079+0000 7f1d16c4f700 1 Processor -- start 2024-01-15T14:35:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d16c4f700 1 -- start start 2024-01-15T14:35:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d16c4f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d10140f20 0x7f1d100b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d16c4f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d10161b40 con 0x7f1d100b2760 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d16c4f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d100b3e10 con 0x7f1d100b3900 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d16c4f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d10161d90 con 0x7f1d10140f20 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d149eb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d10140f20 0x7f1d100b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.080+0000 7f1d151ec700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1d100b2760 0x7f1d1015e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38712/0 (socket says 172.21.15.77:38712) 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d149eb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d10140f20 0x7f1d100b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57718/0 (socket says 172.21.15.77:57718) 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d151ec700 1 -- 172.21.15.77:0/2537239600 learned_addr learned my addr 172.21.15.77:0/2537239600 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d149eb700 1 -- 172.21.15.77:0/2537239600 >> v1:172.21.15.159:6789/0 conn(0x7f1d100b3900 legacy=0x7f1d100b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d149eb700 1 -- 172.21.15.77:0/2537239600 >> v1:172.21.15.77:6789/0 conn(0x7f1d100b2760 legacy=0x7f1d1015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d149eb700 1 -- 172.21.15.77:0/2537239600 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10162070 con 0x7f1d10140f20 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/2537239600 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888686885 0 0) 0x7f1d10161b40 con 0x7f1d100b2760 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d149eb700 1 --2- 172.21.15.77:0/2537239600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d10140f20 0x7f1d100b33c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009840 tx=0x7f1d04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b4efe3550f267d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/2537239600 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04011830 con 0x7f1d10140f20 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/2537239600 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d04011990 con 0x7f1d10140f20 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.081+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/2537239600 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04011d20 con 0x7f1d10140f20 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 -- 172.21.15.77:0/2537239600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d10140f20 msgr2=0x7f1d100b33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/2537239600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d10140f20 0x7f1d100b33c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009840 tx=0x7f1d04009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 -- 172.21.15.77:0/2537239600 shutdown_connections 2024-01-15T14:35:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/2537239600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d10140f20 0x7f1d100b33c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 -- 172.21.15.77:0/2537239600 >> 172.21.15.77:0/2537239600 conn(0x7f1d100743b0 msgr2=0x7f1d1006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 -- 172.21.15.77:0/2537239600 shutdown_connections 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.082+0000 7f1d16c4f700 1 -- 172.21.15.77:0/2537239600 wait complete. 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.083+0000 7f1d16c4f700 1 Processor -- start 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.083+0000 7f1d16c4f700 1 -- start start 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 0x7f1d100aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d100b3900 0x7f1d100af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d10140f20 0x7f1d100b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101f4ed0 con 0x7f1d100b2760 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101f51b0 con 0x7f1d10140f20 2024-01-15T14:35:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101f5490 con 0x7f1d100b3900 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 0x7f1d100aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 0x7f1d100aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57726/0 (socket says 172.21.15.77:57726) 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 -- 172.21.15.77:0/3187547910 learned_addr learned my addr 172.21.15.77:0/3187547910 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d0ffff700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d100b3900 0x7f1d100af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d151ec700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d10140f20 0x7f1d100b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 -- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d100b3900 msgr2=0x7f1d100af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d100b3900 0x7f1d100af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 -- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d10140f20 msgr2=0x7f1d100b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d10140f20 0x7f1d100b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 -- 172.21.15.77:0/3187547910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040094a0 con 0x7f1d100b2760 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d151ec700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d10140f20 0x7f1d100b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d0ffff700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d100b3900 0x7f1d100af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d149eb700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 0x7f1d100aee60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1d04014040 tx=0x7f1d04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0401a980 con 0x7f1d100b2760 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0401ab00 con 0x7f1d100b2760 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d101f5770 con 0x7f1d100b2760 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04024420 con 0x7f1d100b2760 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.084+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d101448f0 con 0x7f1d100b2760 2024-01-15T14:35:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.085+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d10067f60 con 0x7f1d100b2760 2024-01-15T14:35:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.086+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1d04022070 con 0x7f1d100b2760 2024-01-15T14:35:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.086+0000 7f1d0d7fa700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1cf406b380 0x7f1cf406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.086+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d0408ab60 con 0x7f1d100b2760 2024-01-15T14:35:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.086+0000 7f1d0ffff700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1cf406b380 0x7f1cf406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.086+0000 7f1d0ffff700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1cf406b380 0x7f1cf406d840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d101402a0 tx=0x7f1cfc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.091+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d040559e0 con 0x7f1d100b2760 2024-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.280+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d101f5320 con 0x7f1cf406b380 2024-01-15T14:35:14.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.287+0000 7f1d0d7fa700 1 -- 172.21.15.77:0/3187547910 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264572 (secure 0 0 0) 0x7f1d101f5320 con 0x7f1cf406b380 2024-01-15T14:35:14.293 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.290+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1cf406b380 msgr2=0x7f1cf406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.290+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1cf406b380 0x7f1cf406d840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d101402a0 tx=0x7f1cfc009040 comp rx=0 tx=0).stop 2024-01-15T14:35:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.290+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 msgr2=0x7f1d100aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.290+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 0x7f1d100aee60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1d04014040 tx=0x7f1d04004750 comp rx=0 tx=0).stop 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 shutdown_connections 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1cf406b380 0x7f1cf406d840 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d10140f20 0x7f1d100b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100b2760 0x7f1d100aee60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 --2- 172.21.15.77:0/3187547910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d100b3900 0x7f1d100af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 >> 172.21.15.77:0/3187547910 conn(0x7f1d100743b0 msgr2=0x7f1d1006d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 shutdown_connections 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.291+0000 7f1d16c4f700 1 -- 172.21.15.77:0/3187547910 wait complete. 2024-01-15T14:35:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-01-15T14:35:14.304 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-15T14:35:13.021067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003301"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223558+0000","last_change":"2024-01-15T14:35:05.242883+0000","last_active":"2024-01-15T14:35:06.223558+0000","last_peered":"2024-01-15T14:35:06.223558+0000","last_clean":"2024-01-15T14:35:06.223558+0000","last_became_active":"2024-01-15T14:35:05.242650+0000","last_became_peered":"2024-01-15T14:35:05.242650+0000","last_unstale":"2024-01-15T14:35:06.223558+0000","last_undegraded":"2024-01-15T14:35:06.223558+0000","last_fullsized":"2024-01-15T14:35:06.223558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:06:13.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038751+0000","last_change":"2024-01-15T14:35:05.230934+0000","last_active":"2024-01-15T14:35:07.038751+0000","last_peered":"2024-01-15T14:35:07.038751+0000","last_clean":"2024-01-15T14:35:07.038751+0000","last_became_active":"2024-01-15T14:35:05.230356+0000","last_became_peered":"2024-01-15T14:35:05.230356+0000","last_unstale":"2024-01-15T14:35:07.038751+0000","last_undegraded":"2024-01-15T14:35:07.038751+0000","last_fullsized":"2024-01-15T14:35:07.038751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914251+0000","last_change":"2024-01-15T14:35:05.228307+0000","last_active":"2024-01-15T14:35:06.914251+0000","last_peered":"2024-01-15T14:35:06.914251+0000","last_clean":"2024-01-15T14:35:06.914251+0000","last_became_active":"2024-01-15T14:35:05.227876+0000","last_became_peered":"2024-01-15T14:35:05.227876+0000","last_unstale":"2024-01-15T14:35:06.914251+0000","last_undegraded":"2024-01-15T14:35:06.914251+0000","last_fullsized":"2024-01-15T14:35:06.914251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:11.269770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338166+0000","last_change":"2024-01-15T14:35:05.236667+0000","last_active":"2024-01-15T14:35:06.338166+0000","last_peered":"2024-01-15T14:35:06.338166+0000","last_clean":"2024-01-15T14:35:06.338166+0000","last_became_active":"2024-01-15T14:35:05.236363+0000","last_became_peered":"2024-01-15T14:35:05.236363+0000","last_unstale":"2024-01-15T14:35:06.338166+0000","last_undegraded":"2024-01-15T14:35:06.338166+0000","last_fullsized":"2024-01-15T14:35:06.338166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222610+0000","last_change":"2024-01-15T14:35:05.239721+0000","last_active":"2024-01-15T14:35:06.222610+0000","last_peered":"2024-01-15T14:35:06.222610+0000","last_clean":"2024-01-15T14:35:06.222610+0000","last_became_active":"2024-01-15T14:35:05.235648+0000","last_became_peered":"2024-01-15T14:35:05.235648+0000","last_unstale":"2024-01-15T14:35:06.222610+0000","last_undegraded":"2024-01-15T14:35:06.222610+0000","last_fullsized":"2024-01-15T14:35:06.222610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337634+0000","last_change":"2024-01-15T14:35:05.230388+0000","last_active":"2024-01-15T14:35:06.337634+0000","last_peered":"2024-01-15T14:35:06.337634+0000","last_clean":"2024-01-15T14:35:06.337634+0000","last_became_active":"2024-01-15T14:35:05.230109+0000","last_became_peered":"2024-01-15T14:35:05.230109+0000","last_unstale":"2024-01-15T14:35:06.337634+0000","last_undegraded":"2024-01-15T14:35:06.337634+0000","last_fullsized":"2024-01-15T14:35:06.337634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222611+0000","last_change":"2024-01-15T14:35:05.239457+0000","last_active":"2024-01-15T14:35:06.222611+0000","last_peered":"2024-01-15T14:35:06.222611+0000","last_clean":"2024-01-15T14:35:06.222611+0000","last_became_active":"2024-01-15T14:35:05.239152+0000","last_became_peered":"2024-01-15T14:35:05.239152+0000","last_unstale":"2024-01-15T14:35:06.222611+0000","last_undegraded":"2024-01-15T14:35:06.222611+0000","last_fullsized":"2024-01-15T14:35:06.222611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220500+0000","last_change":"2024-01-15T14:35:05.219905+0000","last_active":"2024-01-15T14:35:06.220500+0000","last_peered":"2024-01-15T14:35:06.220500+0000","last_clean":"2024-01-15T14:35:06.220500+0000","last_became_active":"2024-01-15T14:35:05.219331+0000","last_became_peered":"2024-01-15T14:35:05.219331+0000","last_unstale":"2024-01-15T14:35:06.220500+0000","last_undegraded":"2024-01-15T14:35:06.220500+0000","last_fullsized":"2024-01-15T14:35:06.220500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223409+0000","last_change":"2024-01-15T14:35:05.232665+0000","last_active":"2024-01-15T14:35:06.223409+0000","last_peered":"2024-01-15T14:35:06.223409+0000","last_clean":"2024-01-15T14:35:06.223409+0000","last_became_active":"2024-01-15T14:35:05.232381+0000","last_became_peered":"2024-01-15T14:35:05.232381+0000","last_unstale":"2024-01-15T14:35:06.223409+0000","last_undegraded":"2024-01-15T14:35:06.223409+0000","last_fullsized":"2024-01-15T14:35:06.223409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222267+0000","last_change":"2024-01-15T14:35:05.222078+0000","last_active":"2024-01-15T14:35:06.222267+0000","last_peered":"2024-01-15T14:35:06.222267+0000","last_clean":"2024-01-15T14:35:06.222267+0000","last_became_active":"2024-01-15T14:35:05.221809+0000","last_became_peered":"2024-01-15T14:35:05.221809+0000","last_unstale":"2024-01-15T14:35:06.222267+0000","last_undegraded":"2024-01-15T14:35:06.222267+0000","last_fullsized":"2024-01-15T14:35:06.222267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221969+0000","last_change":"2024-01-15T14:35:05.222838+0000","last_active":"2024-01-15T14:35:06.221969+0000","last_peered":"2024-01-15T14:35:06.221969+0000","last_clean":"2024-01-15T14:35:06.221969+0000","last_became_active":"2024-01-15T14:35:05.221973+0000","last_became_peered":"2024-01-15T14:35:05.221973+0000","last_unstale":"2024-01-15T14:35:06.221969+0000","last_undegraded":"2024-01-15T14:35:06.221969+0000","last_fullsized":"2024-01-15T14:35:06.221969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038638+0000","last_change":"2024-01-15T14:35:05.242599+0000","last_active":"2024-01-15T14:35:07.038638+0000","last_peered":"2024-01-15T14:35:07.038638+0000","last_clean":"2024-01-15T14:35:07.038638+0000","last_became_active":"2024-01-15T14:35:05.242171+0000","last_became_peered":"2024-01-15T14:35:05.242171+0000","last_unstale":"2024-01-15T14:35:07.038638+0000","last_undegraded":"2024-01-15T14:35:07.038638+0000","last_fullsized":"2024-01-15T14:35:07.038638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220701+0000","last_change":"2024-01-15T14:35:05.224493+0000","last_active":"2024-01-15T14:35:06.220701+0000","last_peered":"2024-01-15T14:35:06.220701+0000","last_clean":"2024-01-15T14:35:06.220701+0000","last_became_active":"2024-01-15T14:35:05.224166+0000","last_became_peered":"2024-01-15T14:35:05.224166+0000","last_unstale":"2024-01-15T14:35:06.220701+0000","last_undegraded":"2024-01-15T14:35:06.220701+0000","last_fullsized":"2024-01-15T14:35:06.220701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914368+0000","last_change":"2024-01-15T14:35:05.229113+0000","last_active":"2024-01-15T14:35:06.914368+0000","last_peered":"2024-01-15T14:35:06.914368+0000","last_clean":"2024-01-15T14:35:06.914368+0000","last_became_active":"2024-01-15T14:35:05.228790+0000","last_became_peered":"2024-01-15T14:35:05.228790+0000","last_unstale":"2024-01-15T14:35:06.914368+0000","last_undegraded":"2024-01-15T14:35:06.914368+0000","last_fullsized":"2024-01-15T14:35:06.914368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038480+0000","last_change":"2024-01-15T14:35:05.236987+0000","last_active":"2024-01-15T14:35:07.038480+0000","last_peered":"2024-01-15T14:35:07.038480+0000","last_clean":"2024-01-15T14:35:07.038480+0000","last_became_active":"2024-01-15T14:35:05.236693+0000","last_became_peered":"2024-01-15T14:35:05.236693+0000","last_unstale":"2024-01-15T14:35:07.038480+0000","last_undegraded":"2024-01-15T14:35:07.038480+0000","last_fullsized":"2024-01-15T14:35:07.038480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337512+0000","last_change":"2024-01-15T14:35:05.229665+0000","last_active":"2024-01-15T14:35:06.337512+0000","last_peered":"2024-01-15T14:35:06.337512+0000","last_clean":"2024-01-15T14:35:06.337512+0000","last_became_active":"2024-01-15T14:35:05.229391+0000","last_became_peered":"2024-01-15T14:35:05.229391+0000","last_unstale":"2024-01-15T14:35:06.337512+0000","last_undegraded":"2024-01-15T14:35:06.337512+0000","last_fullsized":"2024-01-15T14:35:06.337512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338230+0000","last_change":"2024-01-15T14:35:05.232892+0000","last_active":"2024-01-15T14:35:06.338230+0000","last_peered":"2024-01-15T14:35:06.338230+0000","last_clean":"2024-01-15T14:35:06.338230+0000","last_became_active":"2024-01-15T14:35:05.232596+0000","last_became_peered":"2024-01-15T14:35:05.232596+0000","last_unstale":"2024-01-15T14:35:06.338230+0000","last_undegraded":"2024-01-15T14:35:06.338230+0000","last_fullsized":"2024-01-15T14:35:06.338230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:33.427491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420132+0000","last_change":"2024-01-15T14:35:05.224599+0000","last_active":"2024-01-15T14:35:06.420132+0000","last_peered":"2024-01-15T14:35:06.420132+0000","last_clean":"2024-01-15T14:35:06.420132+0000","last_became_active":"2024-01-15T14:35:05.224306+0000","last_became_peered":"2024-01-15T14:35:05.224306+0000","last_unstale":"2024-01-15T14:35:06.420132+0000","last_undegraded":"2024-01-15T14:35:06.420132+0000","last_fullsized":"2024-01-15T14:35:06.420132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:35.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222848+0000","last_change":"2024-01-15T14:35:05.235613+0000","last_active":"2024-01-15T14:35:06.222848+0000","last_peered":"2024-01-15T14:35:06.222848+0000","last_clean":"2024-01-15T14:35:06.222848+0000","last_became_active":"2024-01-15T14:35:05.235298+0000","last_became_peered":"2024-01-15T14:35:05.235298+0000","last_unstale":"2024-01-15T14:35:06.222848+0000","last_undegraded":"2024-01-15T14:35:06.222848+0000","last_fullsized":"2024-01-15T14:35:06.222848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914462+0000","last_change":"2024-01-15T14:35:04.217677+0000","last_active":"2024-01-15T14:35:06.914462+0000","last_peered":"2024-01-15T14:35:06.914462+0000","last_clean":"2024-01-15T14:35:06.914462+0000","last_became_active":"2024-01-15T14:35:04.217361+0000","last_became_peered":"2024-01-15T14:35:04.217361+0000","last_unstale":"2024-01-15T14:35:06.914462+0000","last_undegraded":"2024-01-15T14:35:06.914462+0000","last_fullsized":"2024-01-15T14:35:06.914462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220970+0000","last_change":"2024-01-15T14:35:05.225192+0000","last_active":"2024-01-15T14:35:06.220970+0000","last_peered":"2024-01-15T14:35:06.220970+0000","last_clean":"2024-01-15T14:35:06.220970+0000","last_became_active":"2024-01-15T14:35:05.224634+0000","last_became_peered":"2024-01-15T14:35:05.224634+0000","last_unstale":"2024-01-15T14:35:06.220970+0000","last_undegraded":"2024-01-15T14:35:06.220970+0000","last_fullsized":"2024-01-15T14:35:06.220970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220874+0000","last_change":"2024-01-15T14:35:04.204838+0000","last_active":"2024-01-15T14:35:06.220874+0000","last_peered":"2024-01-15T14:35:06.220874+0000","last_clean":"2024-01-15T14:35:06.220874+0000","last_became_active":"2024-01-15T14:35:04.203651+0000","last_became_peered":"2024-01-15T14:35:04.203651+0000","last_unstale":"2024-01-15T14:35:06.220874+0000","last_undegraded":"2024-01-15T14:35:06.220874+0000","last_fullsized":"2024-01-15T14:35:06.220874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914437+0000","last_change":"2024-01-15T14:35:05.228513+0000","last_active":"2024-01-15T14:35:06.914437+0000","last_peered":"2024-01-15T14:35:06.914437+0000","last_clean":"2024-01-15T14:35:06.914437+0000","last_became_active":"2024-01-15T14:35:05.228229+0000","last_became_peered":"2024-01-15T14:35:05.228229+0000","last_unstale":"2024-01-15T14:35:06.914437+0000","last_undegraded":"2024-01-15T14:35:06.914437+0000","last_fullsized":"2024-01-15T14:35:06.914437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223203+0000","last_change":"2024-01-15T14:35:04.217602+0000","last_active":"2024-01-15T14:35:06.223203+0000","last_peered":"2024-01-15T14:35:06.223203+0000","last_clean":"2024-01-15T14:35:06.223203+0000","last_became_active":"2024-01-15T14:35:04.217271+0000","last_became_peered":"2024-01-15T14:35:04.217271+0000","last_unstale":"2024-01-15T14:35:06.223203+0000","last_undegraded":"2024-01-15T14:35:06.223203+0000","last_fullsized":"2024-01-15T14:35:06.223203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:24.748909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337912+0000","last_change":"2024-01-15T14:35:05.234884+0000","last_active":"2024-01-15T14:35:06.337912+0000","last_peered":"2024-01-15T14:35:06.337912+0000","last_clean":"2024-01-15T14:35:06.337912+0000","last_became_active":"2024-01-15T14:35:05.234556+0000","last_became_peered":"2024-01-15T14:35:05.234556+0000","last_unstale":"2024-01-15T14:35:06.337912+0000","last_undegraded":"2024-01-15T14:35:06.337912+0000","last_fullsized":"2024-01-15T14:35:06.337912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222800+0000","last_change":"2024-01-15T14:35:04.217549+0000","last_active":"2024-01-15T14:35:06.222800+0000","last_peered":"2024-01-15T14:35:06.222800+0000","last_clean":"2024-01-15T14:35:06.222800+0000","last_became_active":"2024-01-15T14:35:04.217245+0000","last_became_peered":"2024-01-15T14:35:04.217245+0000","last_unstale":"2024-01-15T14:35:06.222800+0000","last_undegraded":"2024-01-15T14:35:06.222800+0000","last_fullsized":"2024-01-15T14:35:06.222800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337934+0000","last_change":"2024-01-15T14:35:05.230835+0000","last_active":"2024-01-15T14:35:06.337934+0000","last_peered":"2024-01-15T14:35:06.337934+0000","last_clean":"2024-01-15T14:35:06.337934+0000","last_became_active":"2024-01-15T14:35:05.230395+0000","last_became_peered":"2024-01-15T14:35:05.230395+0000","last_unstale":"2024-01-15T14:35:06.337934+0000","last_undegraded":"2024-01-15T14:35:06.337934+0000","last_fullsized":"2024-01-15T14:35:06.337934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:06:43.602385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220762+0000","last_change":"2024-01-15T14:35:04.216108+0000","last_active":"2024-01-15T14:35:06.220762+0000","last_peered":"2024-01-15T14:35:06.220762+0000","last_clean":"2024-01-15T14:35:06.220762+0000","last_became_active":"2024-01-15T14:35:04.215774+0000","last_became_peered":"2024-01-15T14:35:04.215774+0000","last_unstale":"2024-01-15T14:35:06.220762+0000","last_undegraded":"2024-01-15T14:35:06.220762+0000","last_fullsized":"2024-01-15T14:35:06.220762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038274+0000","last_change":"2024-01-15T14:35:05.243121+0000","last_active":"2024-01-15T14:35:07.038274+0000","last_peered":"2024-01-15T14:35:07.038274+0000","last_clean":"2024-01-15T14:35:07.038274+0000","last_became_active":"2024-01-15T14:35:05.242858+0000","last_became_peered":"2024-01-15T14:35:05.242858+0000","last_unstale":"2024-01-15T14:35:07.038274+0000","last_undegraded":"2024-01-15T14:35:07.038274+0000","last_fullsized":"2024-01-15T14:35:07.038274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:38.527435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222109+0000","last_change":"2024-01-15T14:35:04.214194+0000","last_active":"2024-01-15T14:35:06.222109+0000","last_peered":"2024-01-15T14:35:06.222109+0000","last_clean":"2024-01-15T14:35:06.222109+0000","last_became_active":"2024-01-15T14:35:04.213952+0000","last_became_peered":"2024-01-15T14:35:04.213952+0000","last_unstale":"2024-01-15T14:35:06.222109+0000","last_undegraded":"2024-01-15T14:35:06.222109+0000","last_fullsized":"2024-01-15T14:35:06.222109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914552+0000","last_change":"2024-01-15T14:35:05.237633+0000","last_active":"2024-01-15T14:35:06.914552+0000","last_peered":"2024-01-15T14:35:06.914552+0000","last_clean":"2024-01-15T14:35:06.914552+0000","last_became_active":"2024-01-15T14:35:05.234559+0000","last_became_peered":"2024-01-15T14:35:05.234559+0000","last_unstale":"2024-01-15T14:35:06.914552+0000","last_undegraded":"2024-01-15T14:35:06.914552+0000","last_fullsized":"2024-01-15T14:35:06.914552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220477+0000","last_change":"2024-01-15T14:35:04.212481+0000","last_active":"2024-01-15T14:35:06.220477+0000","last_peered":"2024-01-15T14:35:06.220477+0000","last_clean":"2024-01-15T14:35:06.220477+0000","last_became_active":"2024-01-15T14:35:04.212205+0000","last_became_peered":"2024-01-15T14:35:04.212205+0000","last_unstale":"2024-01-15T14:35:06.220477+0000","last_undegraded":"2024-01-15T14:35:06.220477+0000","last_fullsized":"2024-01-15T14:35:06.220477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220657+0000","last_change":"2024-01-15T14:35:05.229905+0000","last_active":"2024-01-15T14:35:06.220657+0000","last_peered":"2024-01-15T14:35:06.220657+0000","last_clean":"2024-01-15T14:35:06.220657+0000","last_became_active":"2024-01-15T14:35:05.229591+0000","last_became_peered":"2024-01-15T14:35:05.229591+0000","last_unstale":"2024-01-15T14:35:06.220657+0000","last_undegraded":"2024-01-15T14:35:06.220657+0000","last_fullsized":"2024-01-15T14:35:06.220657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337590+0000","last_change":"2024-01-15T14:35:04.212238+0000","last_active":"2024-01-15T14:35:06.337590+0000","last_peered":"2024-01-15T14:35:06.337590+0000","last_clean":"2024-01-15T14:35:06.337590+0000","last_became_active":"2024-01-15T14:35:04.211931+0000","last_became_peered":"2024-01-15T14:35:04.211931+0000","last_unstale":"2024-01-15T14:35:06.337590+0000","last_undegraded":"2024-01-15T14:35:06.337590+0000","last_fullsized":"2024-01-15T14:35:06.337590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:39.371648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038653+0000","last_change":"2024-01-15T14:35:05.215883+0000","last_active":"2024-01-15T14:35:07.038653+0000","last_peered":"2024-01-15T14:35:07.038653+0000","last_clean":"2024-01-15T14:35:07.038653+0000","last_became_active":"2024-01-15T14:35:05.215584+0000","last_became_peered":"2024-01-15T14:35:05.215584+0000","last_unstale":"2024-01-15T14:35:07.038653+0000","last_undegraded":"2024-01-15T14:35:07.038653+0000","last_fullsized":"2024-01-15T14:35:07.038653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222538+0000","last_change":"2024-01-15T14:35:04.212959+0000","last_active":"2024-01-15T14:35:06.222538+0000","last_peered":"2024-01-15T14:35:06.222538+0000","last_clean":"2024-01-15T14:35:06.222538+0000","last_became_active":"2024-01-15T14:35:04.212632+0000","last_became_peered":"2024-01-15T14:35:04.212632+0000","last_unstale":"2024-01-15T14:35:06.222538+0000","last_undegraded":"2024-01-15T14:35:06.222538+0000","last_fullsized":"2024-01-15T14:35:06.222538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337983+0000","last_change":"2024-01-15T14:35:05.232646+0000","last_active":"2024-01-15T14:35:06.337983+0000","last_peered":"2024-01-15T14:35:06.337983+0000","last_clean":"2024-01-15T14:35:06.337983+0000","last_became_active":"2024-01-15T14:35:05.232360+0000","last_became_peered":"2024-01-15T14:35:05.232360+0000","last_unstale":"2024-01-15T14:35:06.337983+0000","last_undegraded":"2024-01-15T14:35:06.337983+0000","last_fullsized":"2024-01-15T14:35:06.337983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337788+0000","last_change":"2024-01-15T14:35:04.207794+0000","last_active":"2024-01-15T14:35:06.337788+0000","last_peered":"2024-01-15T14:35:06.337788+0000","last_clean":"2024-01-15T14:35:06.337788+0000","last_became_active":"2024-01-15T14:35:04.207503+0000","last_became_peered":"2024-01-15T14:35:04.207503+0000","last_unstale":"2024-01-15T14:35:06.337788+0000","last_undegraded":"2024-01-15T14:35:06.337788+0000","last_fullsized":"2024-01-15T14:35:06.337788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914599+0000","last_change":"2024-01-15T14:35:05.228000+0000","last_active":"2024-01-15T14:35:06.914599+0000","last_peered":"2024-01-15T14:35:06.914599+0000","last_clean":"2024-01-15T14:35:06.914599+0000","last_became_active":"2024-01-15T14:35:05.227713+0000","last_became_peered":"2024-01-15T14:35:05.227713+0000","last_unstale":"2024-01-15T14:35:06.914599+0000","last_undegraded":"2024-01-15T14:35:06.914599+0000","last_fullsized":"2024-01-15T14:35:06.914599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038760+0000","last_change":"2024-01-15T14:35:04.212694+0000","last_active":"2024-01-15T14:35:07.038760+0000","last_peered":"2024-01-15T14:35:07.038760+0000","last_clean":"2024-01-15T14:35:07.038760+0000","last_became_active":"2024-01-15T14:35:04.212368+0000","last_became_peered":"2024-01-15T14:35:04.212368+0000","last_unstale":"2024-01-15T14:35:07.038760+0000","last_undegraded":"2024-01-15T14:35:07.038760+0000","last_fullsized":"2024-01-15T14:35:07.038760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:57.006109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038591+0000","last_change":"2024-01-15T14:35:05.235186+0000","last_active":"2024-01-15T14:35:07.038591+0000","last_peered":"2024-01-15T14:35:07.038591+0000","last_clean":"2024-01-15T14:35:07.038591+0000","last_became_active":"2024-01-15T14:35:05.234887+0000","last_became_peered":"2024-01-15T14:35:05.234887+0000","last_unstale":"2024-01-15T14:35:07.038591+0000","last_undegraded":"2024-01-15T14:35:07.038591+0000","last_fullsized":"2024-01-15T14:35:07.038591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:43.060852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222002+0000","last_change":"2024-01-15T14:35:04.212306+0000","last_active":"2024-01-15T14:35:06.222002+0000","last_peered":"2024-01-15T14:35:06.222002+0000","last_clean":"2024-01-15T14:35:06.222002+0000","last_became_active":"2024-01-15T14:35:04.211871+0000","last_became_peered":"2024-01-15T14:35:04.211871+0000","last_unstale":"2024-01-15T14:35:06.222002+0000","last_undegraded":"2024-01-15T14:35:06.222002+0000","last_fullsized":"2024-01-15T14:35:06.222002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:46.335061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223019+0000","last_change":"2024-01-15T14:35:05.242305+0000","last_active":"2024-01-15T14:35:06.223019+0000","last_peered":"2024-01-15T14:35:06.223019+0000","last_clean":"2024-01-15T14:35:06.223019+0000","last_became_active":"2024-01-15T14:35:05.242026+0000","last_became_peered":"2024-01-15T14:35:05.242026+0000","last_unstale":"2024-01-15T14:35:06.223019+0000","last_undegraded":"2024-01-15T14:35:06.223019+0000","last_fullsized":"2024-01-15T14:35:06.223019+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914403+0000","last_change":"2024-01-15T14:35:04.205419+0000","last_active":"2024-01-15T14:35:06.914403+0000","last_peered":"2024-01-15T14:35:06.914403+0000","last_clean":"2024-01-15T14:35:06.914403+0000","last_became_active":"2024-01-15T14:35:04.205106+0000","last_became_peered":"2024-01-15T14:35:04.205106+0000","last_unstale":"2024-01-15T14:35:06.914403+0000","last_undegraded":"2024-01-15T14:35:06.914403+0000","last_fullsized":"2024-01-15T14:35:06.914403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914366+0000","last_change":"2024-01-15T14:35:05.239426+0000","last_active":"2024-01-15T14:35:06.914366+0000","last_peered":"2024-01-15T14:35:06.914366+0000","last_clean":"2024-01-15T14:35:06.914366+0000","last_became_active":"2024-01-15T14:35:05.239139+0000","last_became_peered":"2024-01-15T14:35:05.239139+0000","last_unstale":"2024-01-15T14:35:06.914366+0000","last_undegraded":"2024-01-15T14:35:06.914366+0000","last_fullsized":"2024-01-15T14:35:06.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338008+0000","last_change":"2024-01-15T14:35:04.207140+0000","last_active":"2024-01-15T14:35:06.338008+0000","last_peered":"2024-01-15T14:35:06.338008+0000","last_clean":"2024-01-15T14:35:06.338008+0000","last_became_active":"2024-01-15T14:35:04.206804+0000","last_became_peered":"2024-01-15T14:35:04.206804+0000","last_unstale":"2024-01-15T14:35:06.338008+0000","last_undegraded":"2024-01-15T14:35:06.338008+0000","last_fullsized":"2024-01-15T14:35:06.338008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:30.291286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222686+0000","last_change":"2024-01-15T14:35:05.240267+0000","last_active":"2024-01-15T14:35:06.222686+0000","last_peered":"2024-01-15T14:35:06.222686+0000","last_clean":"2024-01-15T14:35:06.222686+0000","last_became_active":"2024-01-15T14:35:05.239993+0000","last_became_peered":"2024-01-15T14:35:05.239993+0000","last_unstale":"2024-01-15T14:35:06.222686+0000","last_undegraded":"2024-01-15T14:35:06.222686+0000","last_fullsized":"2024-01-15T14:35:06.222686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221632+0000","last_change":"2024-01-15T14:35:04.214296+0000","last_active":"2024-01-15T14:35:06.221632+0000","last_peered":"2024-01-15T14:35:06.221632+0000","last_clean":"2024-01-15T14:35:06.221632+0000","last_became_active":"2024-01-15T14:35:04.213992+0000","last_became_peered":"2024-01-15T14:35:04.213992+0000","last_unstale":"2024-01-15T14:35:06.221632+0000","last_undegraded":"2024-01-15T14:35:06.221632+0000","last_fullsized":"2024-01-15T14:35:06.221632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221677+0000","last_change":"2024-01-15T14:35:05.231106+0000","last_active":"2024-01-15T14:35:06.221677+0000","last_peered":"2024-01-15T14:35:06.221677+0000","last_clean":"2024-01-15T14:35:06.221677+0000","last_became_active":"2024-01-15T14:35:05.230594+0000","last_became_peered":"2024-01-15T14:35:05.230594+0000","last_unstale":"2024-01-15T14:35:06.221677+0000","last_undegraded":"2024-01-15T14:35:06.221677+0000","last_fullsized":"2024-01-15T14:35:06.221677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:11.015252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420056+0000","last_change":"2024-01-15T14:35:04.211246+0000","last_active":"2024-01-15T14:35:06.420056+0000","last_peered":"2024-01-15T14:35:06.420056+0000","last_clean":"2024-01-15T14:35:06.420056+0000","last_became_active":"2024-01-15T14:35:04.210971+0000","last_became_peered":"2024-01-15T14:35:04.210971+0000","last_unstale":"2024-01-15T14:35:06.420056+0000","last_undegraded":"2024-01-15T14:35:06.420056+0000","last_fullsized":"2024-01-15T14:35:06.420056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:06.847713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222962+0000","last_change":"2024-01-15T14:35:05.240373+0000","last_active":"2024-01-15T14:35:06.222962+0000","last_peered":"2024-01-15T14:35:06.222962+0000","last_clean":"2024-01-15T14:35:06.222962+0000","last_became_active":"2024-01-15T14:35:05.240078+0000","last_became_peered":"2024-01-15T14:35:05.240078+0000","last_unstale":"2024-01-15T14:35:06.222962+0000","last_undegraded":"2024-01-15T14:35:06.222962+0000","last_fullsized":"2024-01-15T14:35:06.222962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:08:59.168691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038437+0000","last_change":"2024-01-15T14:35:04.204121+0000","last_active":"2024-01-15T14:35:07.038437+0000","last_peered":"2024-01-15T14:35:07.038437+0000","last_clean":"2024-01-15T14:35:07.038437+0000","last_became_active":"2024-01-15T14:35:04.203811+0000","last_became_peered":"2024-01-15T14:35:04.203811+0000","last_unstale":"2024-01-15T14:35:07.038437+0000","last_undegraded":"2024-01-15T14:35:07.038437+0000","last_fullsized":"2024-01-15T14:35:07.038437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222934+0000","last_change":"2024-01-15T14:35:05.239960+0000","last_active":"2024-01-15T14:35:06.222934+0000","last_peered":"2024-01-15T14:35:06.222934+0000","last_clean":"2024-01-15T14:35:06.222934+0000","last_became_active":"2024-01-15T14:35:05.239645+0000","last_became_peered":"2024-01-15T14:35:05.239645+0000","last_unstale":"2024-01-15T14:35:06.222934+0000","last_undegraded":"2024-01-15T14:35:06.222934+0000","last_fullsized":"2024-01-15T14:35:06.222934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038563+0000","last_change":"2024-01-15T14:35:04.212055+0000","last_active":"2024-01-15T14:35:07.038563+0000","last_peered":"2024-01-15T14:35:07.038563+0000","last_clean":"2024-01-15T14:35:07.038563+0000","last_became_active":"2024-01-15T14:35:04.211788+0000","last_became_peered":"2024-01-15T14:35:04.211788+0000","last_unstale":"2024-01-15T14:35:07.038563+0000","last_undegraded":"2024-01-15T14:35:07.038563+0000","last_fullsized":"2024-01-15T14:35:07.038563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338047+0000","last_change":"2024-01-15T14:35:05.220108+0000","last_active":"2024-01-15T14:35:06.338047+0000","last_peered":"2024-01-15T14:35:06.338047+0000","last_clean":"2024-01-15T14:35:06.338047+0000","last_became_active":"2024-01-15T14:35:05.219812+0000","last_became_peered":"2024-01-15T14:35:05.219812+0000","last_unstale":"2024-01-15T14:35:06.338047+0000","last_undegraded":"2024-01-15T14:35:06.338047+0000","last_fullsized":"2024-01-15T14:35:06.338047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337937+0000","last_change":"2024-01-15T14:35:04.212432+0000","last_active":"2024-01-15T14:35:06.337937+0000","last_peered":"2024-01-15T14:35:06.337937+0000","last_clean":"2024-01-15T14:35:06.337937+0000","last_became_active":"2024-01-15T14:35:04.211351+0000","last_became_peered":"2024-01-15T14:35:04.211351+0000","last_unstale":"2024-01-15T14:35:06.337937+0000","last_undegraded":"2024-01-15T14:35:06.337937+0000","last_fullsized":"2024-01-15T14:35:06.337937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:38.895254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038566+0000","last_change":"2024-01-15T14:35:05.234938+0000","last_active":"2024-01-15T14:35:07.038566+0000","last_peered":"2024-01-15T14:35:07.038566+0000","last_clean":"2024-01-15T14:35:07.038566+0000","last_became_active":"2024-01-15T14:35:05.234597+0000","last_became_peered":"2024-01-15T14:35:05.234597+0000","last_unstale":"2024-01-15T14:35:07.038566+0000","last_undegraded":"2024-01-15T14:35:07.038566+0000","last_fullsized":"2024-01-15T14:35:07.038566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:00.256458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038392+0000","last_change":"2024-01-15T14:35:05.211465+0000","last_active":"2024-01-15T14:35:07.038392+0000","last_peered":"2024-01-15T14:35:07.038392+0000","last_clean":"2024-01-15T14:35:07.038392+0000","last_became_active":"2024-01-15T14:35:05.211188+0000","last_became_peered":"2024-01-15T14:35:05.211188+0000","last_unstale":"2024-01-15T14:35:07.038392+0000","last_undegraded":"2024-01-15T14:35:07.038392+0000","last_fullsized":"2024-01-15T14:35:07.038392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.419968+0000","last_change":"2024-01-15T14:35:04.208525+0000","last_active":"2024-01-15T14:35:06.419968+0000","last_peered":"2024-01-15T14:35:06.419968+0000","last_clean":"2024-01-15T14:35:06.419968+0000","last_became_active":"2024-01-15T14:35:04.208217+0000","last_became_peered":"2024-01-15T14:35:04.208217+0000","last_unstale":"2024-01-15T14:35:06.419968+0000","last_undegraded":"2024-01-15T14:35:06.419968+0000","last_fullsized":"2024-01-15T14:35:06.419968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:14.822803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220595+0000","last_change":"2024-01-15T14:35:05.236274+0000","last_active":"2024-01-15T14:35:06.220595+0000","last_peered":"2024-01-15T14:35:06.220595+0000","last_clean":"2024-01-15T14:35:06.220595+0000","last_became_active":"2024-01-15T14:35:05.236000+0000","last_became_peered":"2024-01-15T14:35:05.236000+0000","last_unstale":"2024-01-15T14:35:06.220595+0000","last_undegraded":"2024-01-15T14:35:06.220595+0000","last_fullsized":"2024-01-15T14:35:06.220595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337639+0000","last_change":"2024-01-15T14:35:04.209747+0000","last_active":"2024-01-15T14:35:06.337639+0000","last_peered":"2024-01-15T14:35:06.337639+0000","last_clean":"2024-01-15T14:35:06.337639+0000","last_became_active":"2024-01-15T14:35:04.209450+0000","last_became_peered":"2024-01-15T14:35:04.209450+0000","last_unstale":"2024-01-15T14:35:06.337639+0000","last_undegraded":"2024-01-15T14:35:06.337639+0000","last_fullsized":"2024-01-15T14:35:06.337639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:37:47.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T14:35:05.216923+0000","last_change":"2024-01-15T14:34:54.738502+0000","last_active":"2024-01-15T14:35:05.216923+0000","last_peered":"2024-01-15T14:35:05.216923+0000","last_clean":"2024-01-15T14:35:05.216923+0000","last_became_active":"2024-01-15T14:34:54.737354+0000","last_became_peered":"2024-01-15T14:34:54.737354+0000","last_unstale":"2024-01-15T14:35:05.216923+0000","last_undegraded":"2024-01-15T14:35:05.216923+0000","last_fullsized":"2024-01-15T14:35:05.216923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_clean_scrub_stamp":"2024-01-15T14:34:53.124419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:09:14.038173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914075+0000","last_change":"2024-01-15T14:35:05.231021+0000","last_active":"2024-01-15T14:35:06.914075+0000","last_peered":"2024-01-15T14:35:06.914075+0000","last_clean":"2024-01-15T14:35:06.914075+0000","last_became_active":"2024-01-15T14:35:05.230721+0000","last_became_peered":"2024-01-15T14:35:05.230721+0000","last_unstale":"2024-01-15T14:35:06.914075+0000","last_undegraded":"2024-01-15T14:35:06.914075+0000","last_fullsized":"2024-01-15T14:35:06.914075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:15.565484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222435+0000","last_change":"2024-01-15T14:35:04.216271+0000","last_active":"2024-01-15T14:35:06.222435+0000","last_peered":"2024-01-15T14:35:06.222435+0000","last_clean":"2024-01-15T14:35:06.222435+0000","last_became_active":"2024-01-15T14:35:04.215955+0000","last_became_peered":"2024-01-15T14:35:04.215955+0000","last_unstale":"2024-01-15T14:35:06.222435+0000","last_undegraded":"2024-01-15T14:35:06.222435+0000","last_fullsized":"2024-01-15T14:35:06.222435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223126+0000","last_change":"2024-01-15T14:35:04.218008+0000","last_active":"2024-01-15T14:35:06.223126+0000","last_peered":"2024-01-15T14:35:06.223126+0000","last_clean":"2024-01-15T14:35:06.223126+0000","last_became_active":"2024-01-15T14:35:04.217672+0000","last_became_peered":"2024-01-15T14:35:04.217672+0000","last_unstale":"2024-01-15T14:35:06.223126+0000","last_undegraded":"2024-01-15T14:35:06.223126+0000","last_fullsized":"2024-01-15T14:35:06.223126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:25.883807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038622+0000","last_change":"2024-01-15T14:35:05.213577+0000","last_active":"2024-01-15T14:35:07.038622+0000","last_peered":"2024-01-15T14:35:07.038622+0000","last_clean":"2024-01-15T14:35:07.038622+0000","last_became_active":"2024-01-15T14:35:05.212565+0000","last_became_peered":"2024-01-15T14:35:05.212565+0000","last_unstale":"2024-01-15T14:35:07.038622+0000","last_undegraded":"2024-01-15T14:35:07.038622+0000","last_fullsized":"2024-01-15T14:35:07.038622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.106745+0000","last_change":"2024-01-15T14:35:04.216515+0000","last_active":"2024-01-15T14:35:07.106745+0000","last_peered":"2024-01-15T14:35:07.106745+0000","last_clean":"2024-01-15T14:35:07.106745+0000","last_became_active":"2024-01-15T14:35:04.216291+0000","last_became_peered":"2024-01-15T14:35:04.216291+0000","last_unstale":"2024-01-15T14:35:07.106745+0000","last_undegraded":"2024-01-15T14:35:07.106745+0000","last_fullsized":"2024-01-15T14:35:07.106745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:59.252152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221002+0000","last_change":"2024-01-15T14:35:05.233144+0000","last_active":"2024-01-15T14:35:06.221002+0000","last_peered":"2024-01-15T14:35:06.221002+0000","last_clean":"2024-01-15T14:35:06.221002+0000","last_became_active":"2024-01-15T14:35:05.232813+0000","last_became_peered":"2024-01-15T14:35:05.232813+0000","last_unstale":"2024-01-15T14:35:06.221002+0000","last_undegraded":"2024-01-15T14:35:06.221002+0000","last_fullsized":"2024-01-15T14:35:06.221002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:57.453830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222447+0000","last_change":"2024-01-15T14:35:04.215265+0000","last_active":"2024-01-15T14:35:06.222447+0000","last_peered":"2024-01-15T14:35:06.222447+0000","last_clean":"2024-01-15T14:35:06.222447+0000","last_became_active":"2024-01-15T14:35:04.214951+0000","last_became_peered":"2024-01-15T14:35:04.214951+0000","last_unstale":"2024-01-15T14:35:06.222447+0000","last_undegraded":"2024-01-15T14:35:06.222447+0000","last_fullsized":"2024-01-15T14:35:06.222447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914377+0000","last_change":"2024-01-15T14:35:05.228381+0000","last_active":"2024-01-15T14:35:06.914377+0000","last_peered":"2024-01-15T14:35:06.914377+0000","last_clean":"2024-01-15T14:35:06.914377+0000","last_became_active":"2024-01-15T14:35:05.228101+0000","last_became_peered":"2024-01-15T14:35:05.228101+0000","last_unstale":"2024-01-15T14:35:06.914377+0000","last_undegraded":"2024-01-15T14:35:06.914377+0000","last_fullsized":"2024-01-15T14:35:06.914377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222345+0000","last_change":"2024-01-15T14:35:04.215917+0000","last_active":"2024-01-15T14:35:06.222345+0000","last_peered":"2024-01-15T14:35:06.222345+0000","last_clean":"2024-01-15T14:35:06.222345+0000","last_became_active":"2024-01-15T14:35:04.215592+0000","last_became_peered":"2024-01-15T14:35:04.215592+0000","last_unstale":"2024-01-15T14:35:06.222345+0000","last_undegraded":"2024-01-15T14:35:06.222345+0000","last_fullsized":"2024-01-15T14:35:06.222345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:34.320169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038319+0000","last_change":"2024-01-15T14:35:05.233251+0000","last_active":"2024-01-15T14:35:07.038319+0000","last_peered":"2024-01-15T14:35:07.038319+0000","last_clean":"2024-01-15T14:35:07.038319+0000","last_became_active":"2024-01-15T14:35:05.232982+0000","last_became_peered":"2024-01-15T14:35:05.232982+0000","last_unstale":"2024-01-15T14:35:07.038319+0000","last_undegraded":"2024-01-15T14:35:07.038319+0000","last_fullsized":"2024-01-15T14:35:07.038319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:51:15.827108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038435+0000","last_change":"2024-01-15T14:35:04.218539+0000","last_active":"2024-01-15T14:35:07.038435+0000","last_peered":"2024-01-15T14:35:07.038435+0000","last_clean":"2024-01-15T14:35:07.038435+0000","last_became_active":"2024-01-15T14:35:04.218307+0000","last_became_peered":"2024-01-15T14:35:04.218307+0000","last_unstale":"2024-01-15T14:35:07.038435+0000","last_undegraded":"2024-01-15T14:35:07.038435+0000","last_fullsized":"2024-01-15T14:35:07.038435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:42.056546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220625+0000","last_change":"2024-01-15T14:35:05.220403+0000","last_active":"2024-01-15T14:35:06.220625+0000","last_peered":"2024-01-15T14:35:06.220625+0000","last_clean":"2024-01-15T14:35:06.220625+0000","last_became_active":"2024-01-15T14:35:05.220033+0000","last_became_peered":"2024-01-15T14:35:05.220033+0000","last_unstale":"2024-01-15T14:35:06.220625+0000","last_undegraded":"2024-01-15T14:35:06.220625+0000","last_fullsized":"2024-01-15T14:35:06.220625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:31.295592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038582+0000","last_change":"2024-01-15T14:35:04.211061+0000","last_active":"2024-01-15T14:35:07.038582+0000","last_peered":"2024-01-15T14:35:07.038582+0000","last_clean":"2024-01-15T14:35:07.038582+0000","last_became_active":"2024-01-15T14:35:04.210748+0000","last_became_peered":"2024-01-15T14:35:04.210748+0000","last_unstale":"2024-01-15T14:35:07.038582+0000","last_undegraded":"2024-01-15T14:35:07.038582+0000","last_fullsized":"2024-01-15T14:35:07.038582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:37.523130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220819+0000","last_change":"2024-01-15T14:35:05.232247+0000","last_active":"2024-01-15T14:35:06.220819+0000","last_peered":"2024-01-15T14:35:06.220819+0000","last_clean":"2024-01-15T14:35:06.220819+0000","last_became_active":"2024-01-15T14:35:05.231914+0000","last_became_peered":"2024-01-15T14:35:05.231914+0000","last_unstale":"2024-01-15T14:35:06.220819+0000","last_undegraded":"2024-01-15T14:35:06.220819+0000","last_fullsized":"2024-01-15T14:35:06.220819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:25.753215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222720+0000","last_change":"2024-01-15T14:35:04.216586+0000","last_active":"2024-01-15T14:35:06.222720+0000","last_peered":"2024-01-15T14:35:06.222720+0000","last_clean":"2024-01-15T14:35:06.222720+0000","last_became_active":"2024-01-15T14:35:04.216231+0000","last_became_peered":"2024-01-15T14:35:04.216231+0000","last_unstale":"2024-01-15T14:35:06.222720+0000","last_undegraded":"2024-01-15T14:35:06.222720+0000","last_fullsized":"2024-01-15T14:35:06.222720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:31:14.561179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221812+0000","last_change":"2024-01-15T14:35:05.237033+0000","last_active":"2024-01-15T14:35:06.221812+0000","last_peered":"2024-01-15T14:35:06.221812+0000","last_clean":"2024-01-15T14:35:06.221812+0000","last_became_active":"2024-01-15T14:35:05.236738+0000","last_became_peered":"2024-01-15T14:35:05.236738+0000","last_unstale":"2024-01-15T14:35:06.221812+0000","last_undegraded":"2024-01-15T14:35:06.221812+0000","last_fullsized":"2024-01-15T14:35:06.221812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420017+0000","last_change":"2024-01-15T14:35:04.208500+0000","last_active":"2024-01-15T14:35:06.420017+0000","last_peered":"2024-01-15T14:35:06.420017+0000","last_clean":"2024-01-15T14:35:06.420017+0000","last_became_active":"2024-01-15T14:35:04.208227+0000","last_became_peered":"2024-01-15T14:35:04.208227+0000","last_unstale":"2024-01-15T14:35:06.420017+0000","last_undegraded":"2024-01-15T14:35:06.420017+0000","last_fullsized":"2024-01-15T14:35:06.420017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:11:11.486899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223119+0000","last_change":"2024-01-15T14:35:05.232373+0000","last_active":"2024-01-15T14:35:06.223119+0000","last_peered":"2024-01-15T14:35:06.223119+0000","last_clean":"2024-01-15T14:35:06.223119+0000","last_became_active":"2024-01-15T14:35:05.232098+0000","last_became_peered":"2024-01-15T14:35:05.232098+0000","last_unstale":"2024-01-15T14:35:06.223119+0000","last_undegraded":"2024-01-15T14:35:06.223119+0000","last_fullsized":"2024-01-15T14:35:06.223119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:03:29.187189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038754+0000","last_change":"2024-01-15T14:35:04.213296+0000","last_active":"2024-01-15T14:35:07.038754+0000","last_peered":"2024-01-15T14:35:07.038754+0000","last_clean":"2024-01-15T14:35:07.038754+0000","last_became_active":"2024-01-15T14:35:04.212955+0000","last_became_peered":"2024-01-15T14:35:04.212955+0000","last_unstale":"2024-01-15T14:35:07.038754+0000","last_undegraded":"2024-01-15T14:35:07.038754+0000","last_fullsized":"2024-01-15T14:35:07.038754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:10.265464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.338036+0000","last_change":"2024-01-15T14:35:05.219979+0000","last_active":"2024-01-15T14:35:06.338036+0000","last_peered":"2024-01-15T14:35:06.338036+0000","last_clean":"2024-01-15T14:35:06.338036+0000","last_became_active":"2024-01-15T14:35:05.219668+0000","last_became_peered":"2024-01-15T14:35:05.219668+0000","last_unstale":"2024-01-15T14:35:06.338036+0000","last_undegraded":"2024-01-15T14:35:06.338036+0000","last_fullsized":"2024-01-15T14:35:06.338036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:29.761682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420118+0000","last_change":"2024-01-15T14:35:04.208970+0000","last_active":"2024-01-15T14:35:06.420118+0000","last_peered":"2024-01-15T14:35:06.420118+0000","last_clean":"2024-01-15T14:35:06.420118+0000","last_became_active":"2024-01-15T14:35:04.208699+0000","last_became_peered":"2024-01-15T14:35:04.208699+0000","last_unstale":"2024-01-15T14:35:06.420118+0000","last_undegraded":"2024-01-15T14:35:06.420118+0000","last_fullsized":"2024-01-15T14:35:06.420118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:03.181903+0000","last_clean_scrub_stamp":"2024-01-15T14:35:03.181903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:55.457314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.223420+0000","last_change":"2024-01-15T14:35:05.239873+0000","last_active":"2024-01-15T14:35:06.223420+0000","last_peered":"2024-01-15T14:35:06.223420+0000","last_clean":"2024-01-15T14:35:06.223420+0000","last_became_active":"2024-01-15T14:35:05.239571+0000","last_became_peered":"2024-01-15T14:35:05.239571+0000","last_unstale":"2024-01-15T14:35:06.223420+0000","last_undegraded":"2024-01-15T14:35:06.223420+0000","last_fullsized":"2024-01-15T14:35:06.223420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:29:18.138669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.221810+0000","last_change":"2024-01-15T14:35:05.230887+0000","last_active":"2024-01-15T14:35:06.221810+0000","last_peered":"2024-01-15T14:35:06.221810+0000","last_clean":"2024-01-15T14:35:06.221810+0000","last_became_active":"2024-01-15T14:35:05.230187+0000","last_became_peered":"2024-01-15T14:35:05.230187+0000","last_unstale":"2024-01-15T14:35:06.221810+0000","last_undegraded":"2024-01-15T14:35:06.221810+0000","last_fullsized":"2024-01-15T14:35:06.221810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.420110+0000","last_change":"2024-01-15T14:35:05.217285+0000","last_active":"2024-01-15T14:35:06.420110+0000","last_peered":"2024-01-15T14:35:06.420110+0000","last_clean":"2024-01-15T14:35:06.420110+0000","last_became_active":"2024-01-15T14:35:05.217015+0000","last_became_peered":"2024-01-15T14:35:05.217015+0000","last_unstale":"2024-01-15T14:35:06.420110+0000","last_undegraded":"2024-01-15T14:35:06.420110+0000","last_fullsized":"2024-01-15T14:35:06.420110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:47.339366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337918+0000","last_change":"2024-01-15T14:35:05.220986+0000","last_active":"2024-01-15T14:35:06.337918+0000","last_peered":"2024-01-15T14:35:06.337918+0000","last_clean":"2024-01-15T14:35:06.337918+0000","last_became_active":"2024-01-15T14:35:05.220704+0000","last_became_peered":"2024-01-15T14:35:05.220704+0000","last_unstale":"2024-01-15T14:35:06.337918+0000","last_undegraded":"2024-01-15T14:35:06.337918+0000","last_fullsized":"2024-01-15T14:35:06.337918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:41.340152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914367+0000","last_change":"2024-01-15T14:35:05.232048+0000","last_active":"2024-01-15T14:35:06.914367+0000","last_peered":"2024-01-15T14:35:06.914367+0000","last_clean":"2024-01-15T14:35:06.914367+0000","last_became_active":"2024-01-15T14:35:05.231708+0000","last_became_peered":"2024-01-15T14:35:05.231708+0000","last_unstale":"2024-01-15T14:35:06.914367+0000","last_undegraded":"2024-01-15T14:35:06.914367+0000","last_fullsized":"2024-01-15T14:35:06.914367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:39.899559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220904+0000","last_change":"2024-01-15T14:35:05.236304+0000","last_active":"2024-01-15T14:35:06.220904+0000","last_peered":"2024-01-15T14:35:06.220904+0000","last_clean":"2024-01-15T14:35:06.220904+0000","last_became_active":"2024-01-15T14:35:05.235435+0000","last_became_peered":"2024-01-15T14:35:05.235435+0000","last_unstale":"2024-01-15T14:35:06.220904+0000","last_undegraded":"2024-01-15T14:35:06.220904+0000","last_fullsized":"2024-01-15T14:35:06.220904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:41.472205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:07.038669+0000","last_change":"2024-01-15T14:35:05.216092+0000","last_active":"2024-01-15T14:35:07.038669+0000","last_peered":"2024-01-15T14:35:07.038669+0000","last_clean":"2024-01-15T14:35:07.038669+0000","last_became_active":"2024-01-15T14:35:05.215618+0000","last_became_peered":"2024-01-15T14:35:05.215618+0000","last_unstale":"2024-01-15T14:35:07.038669+0000","last_undegraded":"2024-01-15T14:35:07.038669+0000","last_fullsized":"2024-01-15T14:35:07.038669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222044+0000","last_change":"2024-01-15T14:35:05.229045+0000","last_active":"2024-01-15T14:35:06.222044+0000","last_peered":"2024-01-15T14:35:06.222044+0000","last_clean":"2024-01-15T14:35:06.222044+0000","last_became_active":"2024-01-15T14:35:05.228771+0000","last_became_peered":"2024-01-15T14:35:05.228771+0000","last_unstale":"2024-01-15T14:35:06.222044+0000","last_undegraded":"2024-01-15T14:35:06.222044+0000","last_fullsized":"2024-01-15T14:35:06.222044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:00:46.585013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220972+0000","last_change":"2024-01-15T14:35:05.235711+0000","last_active":"2024-01-15T14:35:06.220972+0000","last_peered":"2024-01-15T14:35:06.220972+0000","last_clean":"2024-01-15T14:35:06.220972+0000","last_became_active":"2024-01-15T14:35:05.235419+0000","last_became_peered":"2024-01-15T14:35:05.235419+0000","last_unstale":"2024-01-15T14:35:06.220972+0000","last_undegraded":"2024-01-15T14:35:06.220972+0000","last_fullsized":"2024-01-15T14:35:06.220972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337825+0000","last_change":"2024-01-15T14:35:05.221416+0000","last_active":"2024-01-15T14:35:06.337825+0000","last_peered":"2024-01-15T14:35:06.337825+0000","last_clean":"2024-01-15T14:35:06.337825+0000","last_became_active":"2024-01-15T14:35:05.221124+0000","last_became_peered":"2024-01-15T14:35:05.221124+0000","last_unstale":"2024-01-15T14:35:06.337825+0000","last_undegraded":"2024-01-15T14:35:06.337825+0000","last_fullsized":"2024-01-15T14:35:06.337825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:23.759108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.220561+0000","last_change":"2024-01-15T14:35:05.235785+0000","last_active":"2024-01-15T14:35:06.220561+0000","last_peered":"2024-01-15T14:35:06.220561+0000","last_clean":"2024-01-15T14:35:06.220561+0000","last_became_active":"2024-01-15T14:35:05.235193+0000","last_became_peered":"2024-01-15T14:35:05.235193+0000","last_unstale":"2024-01-15T14:35:06.220561+0000","last_undegraded":"2024-01-15T14:35:06.220561+0000","last_fullsized":"2024-01-15T14:35:06.220561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:31.881202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.337729+0000","last_change":"2024-01-15T14:35:05.225944+0000","last_active":"2024-01-15T14:35:06.337729+0000","last_peered":"2024-01-15T14:35:06.337729+0000","last_clean":"2024-01-15T14:35:06.337729+0000","last_became_active":"2024-01-15T14:35:05.225660+0000","last_became_peered":"2024-01-15T14:35:05.225660+0000","last_unstale":"2024-01-15T14:35:06.337729+0000","last_undegraded":"2024-01-15T14:35:06.337729+0000","last_fullsized":"2024-01-15T14:35:06.337729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:32:22.688652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.914286+0000","last_change":"2024-01-15T14:35:05.243644+0000","last_active":"2024-01-15T14:35:06.914286+0000","last_peered":"2024-01-15T14:35:06.914286+0000","last_clean":"2024-01-15T14:35:06.914286+0000","last_became_active":"2024-01-15T14:35:05.243362+0000","last_became_peered":"2024-01-15T14:35:05.243362+0000","last_unstale":"2024-01-15T14:35:06.914286+0000","last_undegraded":"2024-01-15T14:35:06.914286+0000","last_fullsized":"2024-01-15T14:35:06.914286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:58.010415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T14:35:06.222177+0000","last_change":"2024-01-15T14:35:05.237833+0000","last_active":"2024-01-15T14:35:06.222177+0000","last_peered":"2024-01-15T14:35:06.222177+0000","last_clean":"2024-01-15T14:35:06.222177+0000","last_became_active":"2024-01-15T14:35:05.237576+0000","last_became_peered":"2024-01-15T14:35:05.237576+0000","last_unstale":"2024-01-15T14:35:06.222177+0000","last_undegraded":"2024-01-15T14:35:06.222177+0000","last_fullsized":"2024-01-15T14:35:06.222177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:04.186209+0000","last_clean_scrub_stamp":"2024-01-15T14:35:04.186209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:41:26.888113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9929999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0339999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.106000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.383}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.984999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0650000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4119999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T14:35:14.308 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-15T14:35:14.308 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-15T14:35:14.308 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-15T14:35:14.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-15T14:35:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.462+0000 7f438ac67700 1 Processor -- start 2024-01-15T14:35:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.462+0000 7f438ac67700 1 -- start start 2024-01-15T14:35:14.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f438ac67700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 0x7f438415f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f438ac67700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4384073f90 0x7f4384074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f438ac67700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43840b5c10 con 0x7f43840afb50 2024-01-15T14:35:14.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f438ac67700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438406ee90 con 0x7f4384073f90 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f438ac67700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438415f690 con 0x7f4384161cc0 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4383fff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 0x7f438415f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4383fff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 0x7f438415f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:58120/0 (socket says 172.21.15.77:58120) 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4383fff700 1 -- 172.21.15.77:0/933437803 learned_addr learned my addr 172.21.15.77:0/933437803 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4389204700 1 --2- 172.21.15.77:0/933437803 >> v2:172.21.15.77:3300/0 conn(0x7f4384073f90 0x7f4384074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f43837fe700 1 -- 172.21.15.77:0/933437803 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732207103 0 0) 0x7f43840b5c10 con 0x7f43840afb50 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f43837fe700 1 -- 172.21.15.77:0/933437803 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43740035b0 con 0x7f43840afb50 2024-01-15T14:35:14.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4389204700 1 -- 172.21.15.77:0/933437803 >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 msgr2=0x7f438415f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4389204700 1 --2- 172.21.15.77:0/933437803 >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 0x7f438415f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4389204700 1 -- 172.21.15.77:0/933437803 >> v1:172.21.15.77:6789/0 conn(0x7f43840afb50 legacy=0x7f43840b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4389204700 1 -- 172.21.15.77:0/933437803 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43840748b0 con 0x7f4384073f90 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.463+0000 7f4383fff700 1 --2- 172.21.15.77:0/933437803 >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 0x7f438415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f4389204700 1 --2- 172.21.15.77:0/933437803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384074370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f43840adf90 tx=0x7f437c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98980fa9ec2d720 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f43837fe700 1 -- 172.21.15.77:0/933437803 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00e070 con 0x7f4384073f90 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f43837fe700 1 -- 172.21.15.77:0/933437803 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c00bb20 con 0x7f4384073f90 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f43837fe700 1 -- 172.21.15.77:0/933437803 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c012430 con 0x7f4384073f90 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 -- 172.21.15.77:0/933437803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 msgr2=0x7f4384074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 --2- 172.21.15.77:0/933437803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384074370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f43840adf90 tx=0x7f437c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 -- 172.21.15.77:0/933437803 shutdown_connections 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 --2- 172.21.15.77:0/933437803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384074370 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 --2- 172.21.15.77:0/933437803 >> v2:172.21.15.159:3300/0 conn(0x7f4384161cc0 0x7f438415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 -- 172.21.15.77:0/933437803 >> 172.21.15.77:0/933437803 conn(0x7f4384072a20 msgr2=0x7f438406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 -- 172.21.15.77:0/933437803 shutdown_connections 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.464+0000 7f438ac67700 1 -- 172.21.15.77:0/933437803 wait complete. 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.466+0000 7f438ac67700 1 Processor -- start 2024-01-15T14:35:14.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.466+0000 7f438ac67700 1 -- start start 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f438ac67700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f438ac67700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43840afb50 0x7f4384141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f438ac67700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4384161cc0 0x7f438415b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f438ac67700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438415ab60 con 0x7f4384073f90 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f438ac67700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438415a030 con 0x7f43840afb50 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f438ac67700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438415ae20 con 0x7f4384161cc0 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f4388a03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f4388a03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57752/0 (socket says 172.21.15.77:57752) 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f4388a03700 1 -- 172.21.15.77:0/2805396284 learned_addr learned my addr 172.21.15.77:0/2805396284 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f4383fff700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43840afb50 0x7f4384141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.467+0000 7f4389204700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4384161cc0 0x7f438415b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f4388a03700 1 -- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4384161cc0 msgr2=0x7f438415b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f4388a03700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4384161cc0 0x7f438415b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f4388a03700 1 -- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43840afb50 msgr2=0x7f4384141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f4388a03700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43840afb50 0x7f4384141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f4388a03700 1 -- 172.21.15.77:0/2805396284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370009550 con 0x7f4384073f90 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f4388a03700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384156c60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f437800c520 tx=0x7f437800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43780052d0 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4378005430 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c009a70 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43780038c0 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.468+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4384159a70 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.469+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438415b100 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.469+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4378003a20 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.470+0000 7f43817fa700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f436806b380 0x7f436806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.470+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4378089030 con 0x7f4384073f90 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.470+0000 7f4383fff700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f436806b380 0x7f436806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.470+0000 7f4383fff700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f436806b380 0x7f436806d840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f43840d0210 tx=0x7f43700092c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.474+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4378053e00 con 0x7f4384073f90 2024-01-15T14:35:14.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.716+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4384068f10 con 0x7f4384073f90 2024-01-15T14:35:14.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.716+0000 7f43817fa700 1 -- 172.21.15.77:0/2805396284 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4378017040 con 0x7f4384073f90 2024-01-15T14:35:14.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:14.718 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f436806b380 msgr2=0x7f436806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f436806b380 0x7f436806d840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f43840d0210 tx=0x7f43700092c0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 msgr2=0x7f4384156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384156c60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f437800c520 tx=0x7f437800c830 comp rx=0 tx=0).stop 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 shutdown_connections 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f436806b380 0x7f436806d840 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4384161cc0 0x7f438415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4384073f90 0x7f4384156c60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 --2- 172.21.15.77:0/2805396284 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f43840afb50 0x7f4384141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 >> 172.21.15.77:0/2805396284 conn(0x7f4384072a20 msgr2=0x7f438406bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 shutdown_connections 2024-01-15T14:35:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.718+0000 7f438ac67700 1 -- 172.21.15.77:0/2805396284 wait complete. 2024-01-15T14:35:14.732 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-15T14:35:14.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:14.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.852+0000 7f089a011700 1 Processor -- start 2024-01-15T14:35:14.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.853+0000 7f089a011700 1 -- start start 2024-01-15T14:35:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.853+0000 7f089a011700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f089406c650 0x7f0894165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.853+0000 7f089a011700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089415f800 con 0x7f08940ce140 2024-01-15T14:35:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.853+0000 7f089a011700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08941414c0 con 0x7f089415af10 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.853+0000 7f089a011700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089415c300 con 0x7f089406c650 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.853+0000 7f0893fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f089406c650 0x7f0894165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0893fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f089406c650 0x7f0894165d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57756/0 (socket says 172.21.15.77:57756) 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0893fff700 1 -- 172.21.15.77:0/800800256 learned_addr learned my addr 172.21.15.77:0/800800256 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0892ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f08940ce140 0x7f089415f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38744/0 (socket says 172.21.15.77:38744) 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0893fff700 1 -- 172.21.15.77:0/800800256 >> v1:172.21.15.159:6789/0 conn(0x7f089415af10 legacy=0x7f089415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0893fff700 1 -- 172.21.15.77:0/800800256 >> v1:172.21.15.77:6789/0 conn(0x7f08940ce140 legacy=0x7f089415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f08927fc700 1 -- 172.21.15.77:0/800800256 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212759431 0 0) 0x7f089415f800 con 0x7f08940ce140 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0893fff700 1 -- 172.21.15.77:0/800800256 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08940ce9f0 con 0x7f089406c650 2024-01-15T14:35:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f0893fff700 1 --2- 172.21.15.77:0/800800256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f0894165d70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f088c009e50 tx=0x7f088c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff363ce106d8d59b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f08927fc700 1 -- 172.21.15.77:0/800800256 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00e070 con 0x7f089406c650 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f08927fc700 1 -- 172.21.15.77:0/800800256 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00b9c0 con 0x7f089406c650 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.854+0000 7f08927fc700 1 -- 172.21.15.77:0/800800256 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00bd70 con 0x7f089406c650 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 -- 172.21.15.77:0/800800256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 msgr2=0x7f0894165d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 --2- 172.21.15.77:0/800800256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f0894165d70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f088c009e50 tx=0x7f088c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 -- 172.21.15.77:0/800800256 shutdown_connections 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 --2- 172.21.15.77:0/800800256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f0894165d70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 -- 172.21.15.77:0/800800256 >> 172.21.15.77:0/800800256 conn(0x7f08940cd570 msgr2=0x7f08940cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 -- 172.21.15.77:0/800800256 shutdown_connections 2024-01-15T14:35:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.855+0000 7f089a011700 1 -- 172.21.15.77:0/800800256 wait complete. 2024-01-15T14:35:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.856+0000 7f089a011700 1 Processor -- start 2024-01-15T14:35:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.856+0000 7f089a011700 1 -- start start 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f08940b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08940ce140 0x7f08941577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f089415af10 0x7f089413fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940b3f20 con 0x7f089406c650 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894140490 con 0x7f089415af10 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0894140770 con 0x7f08940ce140 2024-01-15T14:35:14.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f08940b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f08940b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57768/0 (socket says 172.21.15.77:57768) 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 -- 172.21.15.77:0/1965190543 learned_addr learned my addr 172.21.15.77:0/1965190543 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f0893fff700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f089415af10 0x7f089413fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f0892ffd700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08940ce140 0x7f08941577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 -- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08940ce140 msgr2=0x7f08941577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08940ce140 0x7f08941577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 -- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f089415af10 msgr2=0x7f089413fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f089415af10 0x7f089413fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:14.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 -- 172.21.15.77:0/1965190543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c009b30 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f0892ffd700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08940ce140 0x7f08941577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f08937fe700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f08940b43b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f087c00c4e0 tx=0x7f087c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0052d0 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c005430 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08940b38d0 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0038c0 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.857+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0894140080 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.858+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08940b4200 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.859+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f087c003a20 con 0x7f089406c650 2024-01-15T14:35:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.859+0000 7f088bfff700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f087406b350 0x7f087406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.859+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f087c088f60 con 0x7f089406c650 2024-01-15T14:35:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.859+0000 7f0892ffd700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f087406b350 0x7f087406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.860+0000 7f0892ffd700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f087406b350 0x7f087406d810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08941f4500 tx=0x7f0884007d00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:14.864+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f087c053e60 con 0x7f089406c650 2024-01-15T14:35:15.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.052+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f089413d9b0 con 0x7f089406c650 2024-01-15T14:35:15.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.053+0000 7f088bfff700 1 -- 172.21.15.77:0/1965190543 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f087c017090 con 0x7f089406c650 2024-01-15T14:35:15.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:15.055 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f087406b350 msgr2=0x7f087406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f087406b350 0x7f087406d810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08941f4500 tx=0x7f0884007d00 comp rx=0 tx=0).stop 2024-01-15T14:35:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 msgr2=0x7f08940b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f08940b43b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f087c00c4e0 tx=0x7f087c00c7f0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 shutdown_connections 2024-01-15T14:35:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f087406b350 0x7f087406d810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f089415af10 0x7f089413fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089406c650 0x7f08940b43b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 --2- 172.21.15.77:0/1965190543 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f08940ce140 0x7f08941577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 >> 172.21.15.77:0/1965190543 conn(0x7f08940cd570 msgr2=0x7f089415b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 shutdown_connections 2024-01-15T14:35:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.055+0000 7f089a011700 1 -- 172.21.15.77:0/1965190543 wait complete. 2024-01-15T14:35:15.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:15.070 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:15.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.192+0000 7ff268f97700 1 Processor -- start 2024-01-15T14:35:15.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.192+0000 7ff268f97700 1 -- start start 2024-01-15T14:35:15.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff268f97700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff26415e0c0 0x7ff2640afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff268f97700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640b3bf0 0x7ff264073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff268f97700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26415e4a0 con 0x7ff2640744d0 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff268f97700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff264074960 con 0x7ff26415e0c0 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff268f97700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26406aff0 con 0x7ff2640b3bf0 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff26259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff26415e0c0 0x7ff2640afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff26259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff26415e0c0 0x7ff2640afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57770/0 (socket says 172.21.15.77:57770) 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff26259c700 1 -- 172.21.15.77:0/1862879953 learned_addr learned my addr 172.21.15.77:0/1862879953 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff261d9b700 1 --2- 172.21.15.77:0/1862879953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640b3bf0 0x7ff264073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff26259c700 1 -- 172.21.15.77:0/1862879953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640b3bf0 msgr2=0x7ff264073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff26259c700 1 --2- 172.21.15.77:0/1862879953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640b3bf0 0x7ff264073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.193+0000 7ff26259c700 1 -- 172.21.15.77:0/1862879953 >> v1:172.21.15.159:6789/0 conn(0x7ff2640744d0 legacy=0x7ff264161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff26259c700 1 -- 172.21.15.77:0/1862879953 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26406b2d0 con 0x7ff26415e0c0 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff26259c700 1 --2- 172.21.15.77:0/1862879953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff2640afb80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff2580098f0 tx=0x7ff258009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5f20ea97592941 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff26159a700 1 -- 172.21.15.77:0/1862879953 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25800b8a0 con 0x7ff26415e0c0 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff26159a700 1 -- 172.21.15.77:0/1862879953 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25800ba00 con 0x7ff26415e0c0 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff26159a700 1 -- 172.21.15.77:0/1862879953 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25800bd90 con 0x7ff26415e0c0 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 -- 172.21.15.77:0/1862879953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 msgr2=0x7ff2640afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 --2- 172.21.15.77:0/1862879953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff2640afb80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff2580098f0 tx=0x7ff258009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 -- 172.21.15.77:0/1862879953 shutdown_connections 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 --2- 172.21.15.77:0/1862879953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff2640afb80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 --2- 172.21.15.77:0/1862879953 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640b3bf0 0x7ff264073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 -- 172.21.15.77:0/1862879953 >> 172.21.15.77:0/1862879953 conn(0x7ff2640d0c00 msgr2=0x7ff2640d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 -- 172.21.15.77:0/1862879953 shutdown_connections 2024-01-15T14:35:15.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.194+0000 7ff268f97700 1 -- 172.21.15.77:0/1862879953 wait complete. 2024-01-15T14:35:15.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.196+0000 7ff268f97700 1 Processor -- start 2024-01-15T14:35:15.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.196+0000 7ff268f97700 1 -- start start 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.196+0000 7ff268f97700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640744d0 0x7ff2641f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.196+0000 7ff268f97700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2640b3bf0 0x7ff2641553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff268f97700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff264156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff268f97700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2641571d0 con 0x7ff26415e0c0 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff268f97700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2641573b0 con 0x7ff2640b3bf0 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff268f97700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26415c3f0 con 0x7ff2640744d0 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff264156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff264156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57784/0 (socket says 172.21.15.77:57784) 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 -- 172.21.15.77:0/3882642070 learned_addr learned my addr 172.21.15.77:0/3882642070 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff261d9b700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2640b3bf0 0x7ff2641553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff26259c700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640744d0 0x7ff2641f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 -- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640744d0 msgr2=0x7ff2641f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640744d0 0x7ff2641f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 -- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2640b3bf0 msgr2=0x7ff2641553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2640b3bf0 0x7ff2641553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 -- 172.21.15.77:0/3882642070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff258009550 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff261d9b700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2640b3bf0 0x7ff2641553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff262d9d700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff264156c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cb50 tx=0x7ff25c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c003c00 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c003d60 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff264153a10 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00ab10 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.197+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26415bda0 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.198+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26415c5a0 con 0x7ff26415e0c0 2024-01-15T14:35:15.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.199+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff25c00ac70 con 0x7ff26415e0c0 2024-01-15T14:35:15.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.199+0000 7ff252ffd700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff24406b350 0x7ff24406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.199+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff25c089c20 con 0x7ff26415e0c0 2024-01-15T14:35:15.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.199+0000 7ff26259c700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff24406b350 0x7ff24406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.199+0000 7ff26259c700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff24406b350 0x7ff24406d810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff258000c00 tx=0x7ff258008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.203+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff26415c5a0 con 0x7ff26415e0c0 2024-01-15T14:35:15.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.396+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2640b04f0 con 0x7ff26415e0c0 2024-01-15T14:35:15.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.396+0000 7ff252ffd700 1 -- 172.21.15.77:0/3882642070 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff25c054a00 con 0x7ff26415e0c0 2024-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:15.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff24406b350 msgr2=0x7ff24406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff24406b350 0x7ff24406d810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff258000c00 tx=0x7ff258008030 comp rx=0 tx=0).stop 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 msgr2=0x7ff264156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff264156c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cb50 tx=0x7ff25c00cf10 comp rx=0 tx=0).stop 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 shutdown_connections 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff24406b350 0x7ff24406d810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2640744d0 0x7ff2641f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff26415e0c0 0x7ff264156c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 --2- 172.21.15.77:0/3882642070 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2640b3bf0 0x7ff2641553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.398+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 >> 172.21.15.77:0/3882642070 conn(0x7ff2640d0c00 msgr2=0x7ff2640b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.399+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 shutdown_connections 2024-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.399+0000 7ff268f97700 1 -- 172.21.15.77:0/3882642070 wait complete. 2024-01-15T14:35:15.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:15.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.535+0000 7f62d866f700 1 Processor -- start 2024-01-15T14:35:15.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.535+0000 7f62d866f700 1 -- start start 2024-01-15T14:35:15.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.536+0000 7f62d866f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 0x7f62d015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.536+0000 7f62d866f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d00ce870 con 0x7f62d006d3d0 2024-01-15T14:35:15.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.536+0000 7f62d866f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d00b43b0 con 0x7f62d0151d80 2024-01-15T14:35:15.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.536+0000 7f62d866f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0152160 con 0x7f62d00b53d0 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.536+0000 7f62d6c0c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f62d006d3d0 0x7f62d00cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38760/0 (socket says 172.21.15.77:38760) 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.536+0000 7f62d6c0c700 1 -- 172.21.15.77:0/2815688731 learned_addr learned my addr 172.21.15.77:0/2815688731 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5c0a700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 0x7f62d015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304060233 0 0) 0x7f62d00ce870 con 0x7f62d006d3d0 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc0057c0 con 0x7f62d006d3d0 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1172685520 0 0) 0x7f62bc0057c0 con 0x7f62d006d3d0 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 msgr2=0x7f62d015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 0x7f62d015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 >> v1:172.21.15.159:6789/0 conn(0x7f62d0151d80 legacy=0x7f62d00b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d006d820 con 0x7f62d006d3d0 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f62c80019f0 con 0x7f62d006d3d0 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 >> v1:172.21.15.77:6789/0 conn(0x7f62d006d3d0 legacy=0x7f62d00cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62bc005c40 0x7f62bc006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 0x7f62bc006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc006fa0 0x7f62bc007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0152160 con 0x7f62bc005c40 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc00dcd0 con 0x7f62bc006fa0 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc00de90 con 0x7f62d00b53d0 2024-01-15T14:35:15.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62bc005c40 0x7f62bc006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5c0a700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 0x7f62bc006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d6c0c700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc006fa0 0x7f62bc007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 -- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 msgr2=0x7f62bc006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d00b53d0 0x7f62bc006a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 -- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc006fa0 msgr2=0x7f62bc007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc006fa0 0x7f62bc007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 -- 172.21.15.77:0/2815688731 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0057c0 con 0x7f62bc005c40 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d640b700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62bc005c40 0x7f62bc006020 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62c00018d0 tx=0x7f62bc007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70dd3e9bd736cb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.537+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000c690 con 0x7f62bc005c40 2024-01-15T14:35:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c000c810 con 0x7f62bc005c40 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d5409700 1 -- 172.21.15.77:0/2815688731 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000cbc0 con 0x7f62bc005c40 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d866f700 1 -- 172.21.15.77:0/2815688731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62bc005c40 msgr2=0x7f62bc006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d866f700 1 --2- 172.21.15.77:0/2815688731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62bc005c40 0x7f62bc006020 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62c00018d0 tx=0x7f62bc007a90 comp rx=0 tx=0).stop 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d6c0c700 1 -- 172.21.15.77:0/2815688731 reap_dead start 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d866f700 1 -- 172.21.15.77:0/2815688731 shutdown_connections 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d866f700 1 -- 172.21.15.77:0/2815688731 >> 172.21.15.77:0/2815688731 conn(0x7f62d0074170 msgr2=0x7f62d006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d866f700 1 -- 172.21.15.77:0/2815688731 shutdown_connections 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.538+0000 7f62d866f700 1 -- 172.21.15.77:0/2815688731 wait complete. 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.539+0000 7f62d866f700 1 Processor -- start 2024-01-15T14:35:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.539+0000 7f62d866f700 1 -- start start 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d866f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc005c40 0x7f62d0144b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d866f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 0x7f62d01f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d866f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d0155270 0x7f62d0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d866f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0154370 con 0x7f62d0154df0 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d866f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01f46c0 con 0x7f62bc005c40 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d866f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01f49a0 con 0x7f62d0155270 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 0x7f62d01f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 0x7f62d01f3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57798/0 (socket says 172.21.15.77:57798) 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 -- 172.21.15.77:0/2597821142 learned_addr learned my addr 172.21.15.77:0/2597821142 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d640b700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc005c40 0x7f62d0144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d6c0c700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d0155270 0x7f62d0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 -- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d0155270 msgr2=0x7f62d0159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d0155270 0x7f62d0159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 -- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc005c40 msgr2=0x7f62d0144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc005c40 0x7f62d0144b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d5c0a700 1 -- 172.21.15.77:0/2597821142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d01545f0 con 0x7f62d0154df0 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.540+0000 7f62d640b700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc005c40 0x7f62d0144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.541+0000 7f62d5c0a700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 0x7f62d01f3c00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f62cc011660 tx=0x7f62cc011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.541+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc015870 con 0x7f62d0154df0 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.541+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc0159d0 con 0x7f62d0154df0 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.541+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d01548d0 con 0x7f62d0154df0 2024-01-15T14:35:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.541+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc015d40 con 0x7f62d0154df0 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.541+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d00b08c0 con 0x7f62d0154df0 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.542+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0145240 con 0x7f62d0154df0 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.542+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f62cc013400 con 0x7f62d0154df0 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.542+0000 7f62c6ffd700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62b406b350 0x7f62b406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.543+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62cc089f90 con 0x7f62d0154df0 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.543+0000 7f62d640b700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62b406b350 0x7f62b406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.543+0000 7f62d640b700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62b406b350 0x7f62b406d810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62c0014040 tx=0x7f62c0005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.547+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62cc054d60 con 0x7f62d0154df0 2024-01-15T14:35:15.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.741+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62d00753f0 con 0x7f62d0154df0 2024-01-15T14:35:15.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.741+0000 7f62c6ffd700 1 -- 172.21.15.77:0/2597821142 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62cc058380 con 0x7f62d0154df0 2024-01-15T14:35:15.743 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:15.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.743+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62b406b350 msgr2=0x7f62b406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.743+0000 7f62d866f700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62b406b350 0x7f62b406d810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62c0014040 tx=0x7f62c0005040 comp rx=0 tx=0).stop 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.743+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 msgr2=0x7f62d01f3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.743+0000 7f62d866f700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 0x7f62d01f3c00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f62cc011660 tx=0x7f62cc011a20 comp rx=0 tx=0).stop 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.743+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 shutdown_connections 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f62b406b350 0x7f62b406d810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f62d0155270 0x7f62d0159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f62bc005c40 0x7f62d0144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 --2- 172.21.15.77:0/2597821142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d0154df0 0x7f62d01f3c00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 >> 172.21.15.77:0/2597821142 conn(0x7f62d0074170 msgr2=0x7f62d006da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 shutdown_connections 2024-01-15T14:35:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.744+0000 7f62d866f700 1 -- 172.21.15.77:0/2597821142 wait complete. 2024-01-15T14:35:15.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.880+0000 7fc52c642700 1 Processor -- start 2024-01-15T14:35:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.880+0000 7fc52c642700 1 -- start start 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52c642700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc5240afb80 0x7fc524073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52c642700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5240744d0 0x7fc5240add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52c642700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52415e4a0 con 0x7fc52415e0c0 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52c642700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5240b0010 con 0x7fc5240744d0 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52c642700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc524074940 con 0x7fc5240afb80 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5240744d0 0x7fc5240add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5240744d0 0x7fc5240add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57812/0 (socket says 172.21.15.77:57812) 2024-01-15T14:35:15.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 -- 172.21.15.77:0/139523653 learned_addr learned my addr 172.21.15.77:0/139523653 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc529bdd700 1 --2- 172.21.15.77:0/139523653 >> v2:172.21.15.159:3300/0 conn(0x7fc5240afb80 0x7fc524073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 -- 172.21.15.77:0/139523653 >> v2:172.21.15.159:3300/0 conn(0x7fc5240afb80 msgr2=0x7fc524073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 --2- 172.21.15.77:0/139523653 >> v2:172.21.15.159:3300/0 conn(0x7fc5240afb80 0x7fc524073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 -- 172.21.15.77:0/139523653 >> v1:172.21.15.159:6789/0 conn(0x7fc52415e0c0 legacy=0x7fc5240b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 -- 172.21.15.77:0/139523653 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52406aff0 con 0x7fc5240744d0 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc52abdf700 1 --2- 172.21.15.77:0/139523653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240744d0 0x7fc5240add40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009e80 tx=0x7fc51c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96911ce8c7bffc57 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc5293dc700 1 -- 172.21.15.77:0/139523653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c00e070 con 0x7fc5240744d0 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc5293dc700 1 -- 172.21.15.77:0/139523653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c003a20 con 0x7fc5240744d0 2024-01-15T14:35:15.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.881+0000 7fc5293dc700 1 -- 172.21.15.77:0/139523653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c003dd0 con 0x7fc5240744d0 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 -- 172.21.15.77:0/139523653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240744d0 msgr2=0x7fc5240add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 --2- 172.21.15.77:0/139523653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240744d0 0x7fc5240add40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009e80 tx=0x7fc51c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 -- 172.21.15.77:0/139523653 shutdown_connections 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 --2- 172.21.15.77:0/139523653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240744d0 0x7fc5240add40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 --2- 172.21.15.77:0/139523653 >> v2:172.21.15.159:3300/0 conn(0x7fc5240afb80 0x7fc524073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 -- 172.21.15.77:0/139523653 >> 172.21.15.77:0/139523653 conn(0x7fc5240d0c00 msgr2=0x7fc5240d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 -- 172.21.15.77:0/139523653 shutdown_connections 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.882+0000 7fc52c642700 1 -- 172.21.15.77:0/139523653 wait complete. 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.883+0000 7fc52c642700 1 Processor -- start 2024-01-15T14:35:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.883+0000 7fc52c642700 1 -- start start 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc52c642700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5240744d0 0x7fc524153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc52c642700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 0x7fc52415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc52c642700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc52415e0c0 0x7fc52415cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc52c642700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240ce210 con 0x7fc5240afb80 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc52c642700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240cdd80 con 0x7fc52415e0c0 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc52c642700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc524154360 con 0x7fc5240744d0 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc529bdd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 0x7fc52415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc529bdd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 0x7fc52415c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57818/0 (socket says 172.21.15.77:57818) 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.884+0000 7fc529bdd700 1 -- 172.21.15.77:0/2101194886 learned_addr learned my addr 172.21.15.77:0/2101194886 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc52abdf700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc52415e0c0 0x7fc52415cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc52a3de700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5240744d0 0x7fc524153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc529bdd700 1 -- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5240744d0 msgr2=0x7fc524153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc529bdd700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5240744d0 0x7fc524153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc529bdd700 1 -- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc52415e0c0 msgr2=0x7fc52415cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc529bdd700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc52415e0c0 0x7fc52415cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc529bdd700 1 -- 172.21.15.77:0/2101194886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c009b30 con 0x7fc5240afb80 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc52a3de700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5240744d0 0x7fc524153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:15.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc529bdd700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 0x7fc52415c700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc51400c7f0 tx=0x7fc51400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514011800 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc514011960 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5240cd170 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514011cf0 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.885+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5240cd730 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.886+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5240cdef0 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.887+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc514024480 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.887+0000 7fc51affd700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50806b350 0x7fc50806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.887+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc514089fe0 con 0x7fc5240afb80 2024-01-15T14:35:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.887+0000 7fc52a3de700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50806b350 0x7fc50806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:15.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.887+0000 7fc52a3de700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50806b350 0x7fc50806d810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc520001750 tx=0x7fc520009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:15.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:15.891+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc514054ee0 con 0x7fc5240afb80 2024-01-15T14:35:16.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.079+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc524069020 con 0x7fc5240afb80 2024-01-15T14:35:16.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.079+0000 7fc51affd700 1 -- 172.21.15.77:0/2101194886 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc51401f0a0 con 0x7fc5240afb80 2024-01-15T14:35:16.081 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:16.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50806b350 msgr2=0x7fc50806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50806b350 0x7fc50806d810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc520001750 tx=0x7fc520009040 comp rx=0 tx=0).stop 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 msgr2=0x7fc52415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 0x7fc52415c700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc51400c7f0 tx=0x7fc51400cbb0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 shutdown_connections 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50806b350 0x7fc50806d810 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc5240744d0 0x7fc524153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc52415e0c0 0x7fc52415cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 --2- 172.21.15.77:0/2101194886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5240afb80 0x7fc52415c700 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 >> 172.21.15.77:0/2101194886 conn(0x7fc5240d0c00 msgr2=0x7fc52406dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:16.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 shutdown_connections 2024-01-15T14:35:16.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.082+0000 7fc52c642700 1 -- 172.21.15.77:0/2101194886 wait complete. 2024-01-15T14:35:16.095 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.215+0000 7f3745083700 1 Processor -- start 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.215+0000 7f3745083700 1 -- start start 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f3745083700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37400ce090 0x7f374006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f3745083700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3740073f90 0x7f3740074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f3745083700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374006e7f0 con 0x7f374015e2a0 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f3745083700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3740074960 con 0x7f37400ce090 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f3745083700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374006dec0 con 0x7f3740073f90 2024-01-15T14:35:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37400ce090 0x7f374006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37400ce090 0x7f374006ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57826/0 (socket says 172.21.15.77:57826) 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 -- 172.21.15.77:0/3998189193 learned_addr learned my addr 172.21.15.77:0/3998189193 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373f59e700 1 --2- 172.21.15.77:0/3998189193 >> v2:172.21.15.159:3300/0 conn(0x7f3740073f90 0x7f3740074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 -- 172.21.15.77:0/3998189193 >> v2:172.21.15.159:3300/0 conn(0x7f3740073f90 msgr2=0x7f3740074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 --2- 172.21.15.77:0/3998189193 >> v2:172.21.15.159:3300/0 conn(0x7f3740073f90 0x7f3740074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 -- 172.21.15.77:0/3998189193 >> v1:172.21.15.159:6789/0 conn(0x7f374015e2a0 legacy=0x7f374006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 -- 172.21.15.77:0/3998189193 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374006e1a0 con 0x7f37400ce090 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373e59c700 1 --2- 172.21.15.77:0/3998189193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37400ce090 0x7f374006ec80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f37400b5c40 tx=0x7f3734009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9a122c2fd66768 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373dd9b700 1 -- 172.21.15.77:0/3998189193 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734004230 con 0x7f37400ce090 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373dd9b700 1 -- 172.21.15.77:0/3998189193 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3734004390 con 0x7f37400ce090 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.216+0000 7f373dd9b700 1 -- 172.21.15.77:0/3998189193 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37340036a0 con 0x7f37400ce090 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 -- 172.21.15.77:0/3998189193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37400ce090 msgr2=0x7f374006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 --2- 172.21.15.77:0/3998189193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37400ce090 0x7f374006ec80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f37400b5c40 tx=0x7f3734009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 -- 172.21.15.77:0/3998189193 shutdown_connections 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 --2- 172.21.15.77:0/3998189193 >> v2:172.21.15.159:3300/0 conn(0x7f3740073f90 0x7f3740074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 --2- 172.21.15.77:0/3998189193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37400ce090 0x7f374006ec80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 -- 172.21.15.77:0/3998189193 >> 172.21.15.77:0/3998189193 conn(0x7f37400cd570 msgr2=0x7f37400cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 -- 172.21.15.77:0/3998189193 shutdown_connections 2024-01-15T14:35:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.217+0000 7f3745083700 1 -- 172.21.15.77:0/3998189193 wait complete. 2024-01-15T14:35:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.218+0000 7f3745083700 1 Processor -- start 2024-01-15T14:35:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.218+0000 7f3745083700 1 -- start start 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f3745083700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 0x7f37400c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f3745083700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37400ce090 0x7f37400af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f3745083700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f374015e2a0 0x7f37400c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f3745083700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3740150c50 con 0x7f3740073f90 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f3745083700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374015f970 con 0x7f37400ce090 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f3745083700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374015ed60 con 0x7f374015e2a0 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 0x7f37400c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 0x7f37400c5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57832/0 (socket says 172.21.15.77:57832) 2024-01-15T14:35:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373ed9d700 1 -- 172.21.15.77:0/84578761 learned_addr learned my addr 172.21.15.77:0/84578761 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373e59c700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37400ce090 0x7f37400af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373f59e700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f374015e2a0 0x7f37400c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373ed9d700 1 -- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f374015e2a0 msgr2=0x7f37400c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373ed9d700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f374015e2a0 0x7f37400c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.219+0000 7f373ed9d700 1 -- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37400ce090 msgr2=0x7f37400af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f373ed9d700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37400ce090 0x7f37400af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f373ed9d700 1 -- 172.21.15.77:0/84578761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3734009550 con 0x7f3740073f90 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f373ed9d700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 0x7f37400c5a50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f373000c4a0 tx=0x7f373000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300054e0 con 0x7f3740073f90 2024-01-15T14:35:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3730003680 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730003a30 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374015f040 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.220+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374015f600 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.221+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374015f790 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.221+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3730003b90 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.221+0000 7f37277fe700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f372c06b350 0x7f372c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.222+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3730088b10 con 0x7f3740073f90 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.222+0000 7f373e59c700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f372c06b350 0x7f372c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.222+0000 7f373e59c700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f372c06b350 0x7f372c06d810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3734009130 tx=0x7f3734008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.226+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3730053980 con 0x7f3740073f90 2024-01-15T14:35:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.461+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f374006f190 con 0x7f3740073f90 2024-01-15T14:35:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.462+0000 7f37277fe700 1 -- 172.21.15.77:0/84578761 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f3730056fa0 con 0x7f3740073f90 2024-01-15T14:35:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:16.463 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f372c06b350 msgr2=0x7f372c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f372c06b350 0x7f372c06d810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3734009130 tx=0x7f3734008030 comp rx=0 tx=0).stop 2024-01-15T14:35:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 msgr2=0x7f37400c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 0x7f37400c5a50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f373000c4a0 tx=0x7f373000c7b0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 shutdown_connections 2024-01-15T14:35:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f372c06b350 0x7f372c06d810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f374015e2a0 0x7f37400c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3740073f90 0x7f37400c5a50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 --2- 172.21.15.77:0/84578761 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f37400ce090 0x7f37400af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 >> 172.21.15.77:0/84578761 conn(0x7f37400cd570 msgr2=0x7f37400cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 shutdown_connections 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.464+0000 7f3745083700 1 -- 172.21.15.77:0/84578761 wait complete. 2024-01-15T14:35:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:16.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:16.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.598+0000 7f19632f8700 1 Processor -- start 2024-01-15T14:35:16.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.598+0000 7f19632f8700 1 -- start start 2024-01-15T14:35:16.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f19632f8700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f195c158a50 0x7f195c0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f19632f8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0cfff0 con 0x7f195c158670 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f19632f8700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0cfd10 con 0x7f195c162a30 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f19632f8700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c0629b0 con 0x7f195c158a50 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1961094700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f195c158670 0x7f195c0622a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38772/0 (socket says 172.21.15.77:38772) 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1961094700 1 -- 172.21.15.77:0/1714807720 learned_addr learned my addr 172.21.15.77:0/1714807720 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322377522 0 0) 0x7f195c0cfff0 con 0x7f195c158670 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19480035b0 con 0x7f195c158670 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1960893700 1 --2- 172.21.15.77:0/1714807720 >> v2:172.21.15.159:3300/0 conn(0x7f195c158a50 0x7f195c0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 342231560 0 0) 0x7f19480035b0 con 0x7f195c158670 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 >> v2:172.21.15.159:3300/0 conn(0x7f195c158a50 msgr2=0x7f195c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 --2- 172.21.15.77:0/1714807720 >> v2:172.21.15.159:3300/0 conn(0x7f195c158a50 0x7f195c0c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 >> v1:172.21.15.159:6789/0 conn(0x7f195c162a30 legacy=0x7f195c0ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c161f50 con 0x7f195c158670 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1960893700 1 --2- 172.21.15.77:0/1714807720 >> v2:172.21.15.159:3300/0 conn(0x7f195c158a50 0x7f195c0c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f1958001670 con 0x7f195c158670 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 >> v1:172.21.15.77:6789/0 conn(0x7f195c158670 legacy=0x7f195c0622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f1948003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1948004850 0x7f1948004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1948005410 0x7f194800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0629b0 con 0x7f1948003a00 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194800c2a0 con 0x7f1948004850 2024-01-15T14:35:16.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.599+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194800c460 con 0x7f1948005410 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f1948003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1960893700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1948004850 0x7f1948004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961895700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1948005410 0x7f194800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 -- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1948005410 msgr2=0x7f194800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1948005410 0x7f194800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 -- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1948004850 msgr2=0x7f1948004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1948004850 0x7f1948004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 -- 172.21.15.77:0/1714807720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480035b0 con 0x7f1948003a00 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961895700 1 -- 172.21.15.77:0/1714807720 reap_dead start 2024-01-15T14:35:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1961094700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f1948003e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f195800c240 tx=0x7f1948005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c090df1bcfbf5e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958007c10 con 0x7f1948003a00 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195800ce00 con 0x7f1948003a00 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f1953fff700 1 -- 172.21.15.77:0/1714807720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958005ba0 con 0x7f1948003a00 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f19632f8700 1 -- 172.21.15.77:0/1714807720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 msgr2=0x7f1948003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.600+0000 7f19632f8700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f1948003e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f195800c240 tx=0x7f1948005880 comp rx=0 tx=0).stop 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.601+0000 7f19632f8700 1 -- 172.21.15.77:0/1714807720 shutdown_connections 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.601+0000 7f19632f8700 1 --2- 172.21.15.77:0/1714807720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f1948003e20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.601+0000 7f19632f8700 1 -- 172.21.15.77:0/1714807720 >> 172.21.15.77:0/1714807720 conn(0x7f195c0b57b0 msgr2=0x7f195c0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.601+0000 7f19632f8700 1 -- 172.21.15.77:0/1714807720 shutdown_connections 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.601+0000 7f19632f8700 1 -- 172.21.15.77:0/1714807720 wait complete. 2024-01-15T14:35:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 Processor -- start 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 -- start start 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f195c0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f195c160ab0 0x7f195c0cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f195c1f54d0 0x7f195c1f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c14bd40 con 0x7f1948003a00 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.602+0000 7f19632f8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0cda30 con 0x7f195c1f54d0 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f19632f8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c160010 con 0x7f195c160ab0 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f195c0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f195c0cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57850/0 (socket says 172.21.15.77:57850) 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 -- 172.21.15.77:0/1754539217 learned_addr learned my addr 172.21.15.77:0/1754539217 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961895700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f195c1f54d0 0x7f195c1f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 -- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f195c160ab0 msgr2=0x7f195c0cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1960893700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f195c160ab0 0x7f195c0cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f195c160ab0 0x7f195c0cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 -- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f195c1f54d0 msgr2=0x7f195c1f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f195c1f54d0 0x7f195c1f5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 -- 172.21.15.77:0/1754539217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0cd750 con 0x7f1948003a00 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961895700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f195c1f54d0 0x7f195c1f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1961094700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f195c0cd150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f194800c270 tx=0x7f1958001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958001630 con 0x7f1948003a00 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19580070d0 con 0x7f1948003a00 2024-01-15T14:35:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c1602b0 con 0x7f1948003a00 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958007480 con 0x7f1948003a00 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.603+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c0cbae0 con 0x7f1948003a00 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.604+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c06d070 con 0x7f1948003a00 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.605+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f19580075e0 con 0x7f1948003a00 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.605+0000 7f1951ffb700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f194006b350 0x7f194006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.605+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f195808b340 con 0x7f1948003a00 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.605+0000 7f1960893700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f194006b350 0x7f194006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.606+0000 7f1960893700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f194006b350 0x7f194006d810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f195c0cb100 tx=0x7f194c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.610+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1958056360 con 0x7f1948003a00 2024-01-15T14:35:16.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.799+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f195c068670 con 0x7f1948003a00 2024-01-15T14:35:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.800+0000 7f1951ffb700 1 -- 172.21.15.77:0/1754539217 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1958059980 con 0x7f1948003a00 2024-01-15T14:35:16.802 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:16.802 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:16.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f194006b350 msgr2=0x7f194006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f194006b350 0x7f194006d810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f195c0cb100 tx=0x7f194c008040 comp rx=0 tx=0).stop 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 msgr2=0x7f195c0cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f195c0cd150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f194800c270 tx=0x7f1958001a00 comp rx=0 tx=0).stop 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 shutdown_connections 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f194006b350 0x7f194006d810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f195c1f54d0 0x7f195c1f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1948003a00 0x7f195c0cd150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 --2- 172.21.15.77:0/1754539217 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f195c160ab0 0x7f195c0cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 >> 172.21.15.77:0/1754539217 conn(0x7f195c0b57b0 msgr2=0x7f195c0c98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 shutdown_connections 2024-01-15T14:35:16.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.802+0000 7f19632f8700 1 -- 172.21.15.77:0/1754539217 wait complete. 2024-01-15T14:35:16.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:16.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.939+0000 7fbcccfb4700 1 Processor -- start 2024-01-15T14:35:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.939+0000 7fbcccfb4700 1 -- start start 2024-01-15T14:35:16.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcccfb4700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 0x7fbcc80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcccfb4700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fbcc8073f90 0x7fbcc806e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcccfb4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc80b42a0 con 0x7fbcc815e280 2024-01-15T14:35:16.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcccfb4700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc80af900 con 0x7fbcc8073f90 2024-01-15T14:35:16.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcccfb4700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8074d30 con 0x7fbcc80b5220 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcc659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbcc815e280 0x7fbcc80b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38780/0 (socket says 172.21.15.77:38780) 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.940+0000 7fbcc659c700 1 -- 172.21.15.77:0/3748918933 learned_addr learned my addr 172.21.15.77:0/3748918933 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc5d9b700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 0x7fbcc80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc6d9d700 1 --2- 172.21.15.77:0/3748918933 >> v2:172.21.15.159:3300/0 conn(0x7fbcc8073f90 0x7fbcc806e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631443000 0 0) 0x7fbcbc001110 con 0x7fbcc815e280 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb00035b0 con 0x7fbcc815e280 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2422622659 0 0) 0x7fbcb00035b0 con 0x7fbcc815e280 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 msgr2=0x7fbcc80b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 0x7fbcc80b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 >> v2:172.21.15.159:3300/0 conn(0x7fbcc8073f90 msgr2=0x7fbcc806e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 --2- 172.21.15.77:0/3748918933 >> v2:172.21.15.159:3300/0 conn(0x7fbcc8073f90 0x7fbcc806e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc8074490 con 0x7fbcc815e280 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fbcbc001800 con 0x7fbcc815e280 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 >> v1:172.21.15.77:6789/0 conn(0x7fbcc815e280 legacy=0x7fbcc80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb0003a00 0x7fbcb0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 0x7fbcb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0004d90 0x7fbcb0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80af900 con 0x7fbcb0003a00 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8074d30 con 0x7fbcb0004d90 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb000ba10 con 0x7fbcc80b5220 2024-01-15T14:35:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb0003a00 0x7fbcb0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc6d9d700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0004d90 0x7fbcb0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc5d9b700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 0x7fbcb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 -- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 msgr2=0x7fbcb0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80b5220 0x7fbcb0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 -- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0004d90 msgr2=0x7fbcb0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0004d90 0x7fbcb0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 -- 172.21.15.77:0/3748918933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb00035b0 con 0x7fbcb0003a00 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.941+0000 7fbcc659c700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb0003a00 0x7fbcb0003e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00c2f0 tx=0x7fbcb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6d94b494c701c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc00ce30 con 0x7fbcb0003a00 2024-01-15T14:35:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc005960 con 0x7fbcb0003a00 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcc559a700 1 -- 172.21.15.77:0/3748918933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc005d10 con 0x7fbcb0003a00 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcccfb4700 1 -- 172.21.15.77:0/3748918933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb0003a00 msgr2=0x7fbcb0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/3748918933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb0003a00 0x7fbcb0003e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00c2f0 tx=0x7fbcb0005880 comp rx=0 tx=0).stop 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcc6d9d700 1 -- 172.21.15.77:0/3748918933 reap_dead start 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcccfb4700 1 -- 172.21.15.77:0/3748918933 shutdown_connections 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcccfb4700 1 -- 172.21.15.77:0/3748918933 >> 172.21.15.77:0/3748918933 conn(0x7fbcc8072b20 msgr2=0x7fbcc806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcccfb4700 1 -- 172.21.15.77:0/3748918933 shutdown_connections 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.942+0000 7fbcccfb4700 1 -- 172.21.15.77:0/3748918933 wait complete. 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.943+0000 7fbcccfb4700 1 Processor -- start 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 -- start start 2024-01-15T14:35:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0003a00 0x7fbcc8157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 0x7fbcc8148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80ce4e0 0x7fbcc80ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80ceea0 con 0x7fbcc8153f50 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80cfda0 con 0x7fbcb0003a00 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcccfb4700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8148f70 con 0x7fbcc80ce4e0 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcc5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 0x7fbcc8148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcc5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 0x7fbcc8148670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57872/0 (socket says 172.21.15.77:57872) 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.944+0000 7fbcc5d9b700 1 -- 172.21.15.77:0/1972346387 learned_addr learned my addr 172.21.15.77:0/1972346387 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc659c700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0003a00 0x7fbcc8157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc6d9d700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80ce4e0 0x7fbcc80ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc5d9b700 1 -- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80ce4e0 msgr2=0x7fbcc80ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc5d9b700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80ce4e0 0x7fbcc80ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc5d9b700 1 -- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0003a00 msgr2=0x7fbcc8157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc5d9b700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0003a00 0x7fbcc8157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc5d9b700 1 -- 172.21.15.77:0/1972346387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc81577a0 con 0x7fbcc8153f50 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc659c700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0003a00 0x7fbcc8157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcc5d9b700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 0x7fbcc8148670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8011ad0 tx=0x7fbcb8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800fa00 con 0x7fbcc8153f50 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb800fb60 con 0x7fbcc8153f50 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc8148bb0 con 0x7fbcc8153f50 2024-01-15T14:35:16.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8014440 con 0x7fbcc8153f50 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.945+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc8157d40 con 0x7fbcc8153f50 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.946+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc81492a0 con 0x7fbcc8153f50 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.946+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbcb80145a0 con 0x7fbcc8153f50 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.947+0000 7fbcb6ffd700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbca806b350 0x7fbca806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.947+0000 7fbcc659c700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbca806b350 0x7fbca806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.947+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcb808a350 con 0x7fbcc8153f50 2024-01-15T14:35:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.947+0000 7fbcc659c700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbca806b350 0x7fbca806d810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc008fd0 tx=0x7fbcbc00a000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:16.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:16.951+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbcb8055280 con 0x7fbcc8153f50 2024-01-15T14:35:17.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.142+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbcc8069020 con 0x7fbcc8153f50 2024-01-15T14:35:17.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.142+0000 7fbcb6ffd700 1 -- 172.21.15.77:0/1972346387 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbcb80588a0 con 0x7fbcc8153f50 2024-01-15T14:35:17.143 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:17.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.144+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbca806b350 msgr2=0x7fbca806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.144+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbca806b350 0x7fbca806d810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc008fd0 tx=0x7fbcbc00a000 comp rx=0 tx=0).stop 2024-01-15T14:35:17.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 msgr2=0x7fbcc8148670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 0x7fbcc8148670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8011ad0 tx=0x7fbcb8011e90 comp rx=0 tx=0).stop 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 shutdown_connections 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fbca806b350 0x7fbca806d810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbcc80ce4e0 0x7fbcc80ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbcb0003a00 0x7fbcc8157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 --2- 172.21.15.77:0/1972346387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcc8153f50 0x7fbcc8148670 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 >> 172.21.15.77:0/1972346387 conn(0x7fbcc8072b20 msgr2=0x7fbcc806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 shutdown_connections 2024-01-15T14:35:17.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.145+0000 7fbcccfb4700 1 -- 172.21.15.77:0/1972346387 wait complete. 2024-01-15T14:35:17.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:17.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.281+0000 7f96619c4700 1 Processor -- start 2024-01-15T14:35:17.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.281+0000 7f96619c4700 1 -- start start 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f96619c4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f965c06dae0 0x7f965c15c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f96619c4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c06def0 con 0x7f965c0adf60 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f96619c4700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c0b05e0 con 0x7f965c06c000 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f96619c4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965c0af960 con 0x7f965c06dae0 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f965b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f965c06dae0 0x7f965c15c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f965a7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f965c0adf60 0x7f965c06f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38784/0 (socket says 172.21.15.77:38784) 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f965a7fc700 1 -- 172.21.15.77:0/4062622382 learned_addr learned my addr 172.21.15.77:0/4062622382 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f965b7fe700 1 -- 172.21.15.77:0/4062622382 >> v1:172.21.15.159:6789/0 conn(0x7f965c06c000 legacy=0x7f965c06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f965b7fe700 1 -- 172.21.15.77:0/4062622382 >> v1:172.21.15.77:6789/0 conn(0x7f965c0adf60 legacy=0x7f965c06f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f9659ffb700 1 -- 172.21.15.77:0/4062622382 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191548964 0 0) 0x7f964c001110 con 0x7f965c0adf60 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.282+0000 7f965b7fe700 1 -- 172.21.15.77:0/4062622382 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c0afc40 con 0x7f965c06dae0 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f965b7fe700 1 --2- 172.21.15.77:0/4062622382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15c760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9654009d90 tx=0x7f965400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4ebd0b3782ab50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f9659ffb700 1 -- 172.21.15.77:0/4062622382 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965400e070 con 0x7f965c06dae0 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f9659ffb700 1 -- 172.21.15.77:0/4062622382 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965400bbd0 con 0x7f965c06dae0 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f9659ffb700 1 -- 172.21.15.77:0/4062622382 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9654012430 con 0x7f965c06dae0 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 -- 172.21.15.77:0/4062622382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 msgr2=0x7f965c15c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 --2- 172.21.15.77:0/4062622382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15c760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9654009d90 tx=0x7f965400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 -- 172.21.15.77:0/4062622382 shutdown_connections 2024-01-15T14:35:17.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 --2- 172.21.15.77:0/4062622382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15c760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 -- 172.21.15.77:0/4062622382 >> 172.21.15.77:0/4062622382 conn(0x7f965c071bb0 msgr2=0x7f965c071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:17.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 -- 172.21.15.77:0/4062622382 shutdown_connections 2024-01-15T14:35:17.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.283+0000 7f96619c4700 1 -- 172.21.15.77:0/4062622382 wait complete. 2024-01-15T14:35:17.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.284+0000 7f96619c4700 1 Processor -- start 2024-01-15T14:35:17.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 -- start start 2024-01-15T14:35:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c06c000 0x7f965c151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c0adf60 0x7f965c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c152000 con 0x7f965c06dae0 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c1e4260 con 0x7f965c0adf60 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.285+0000 7f96619c4700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c1e4540 con 0x7f965c06c000 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57894/0 (socket says 172.21.15.77:57894) 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 -- 172.21.15.77:0/4048956027 learned_addr learned my addr 172.21.15.77:0/4048956027 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965b7fe700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c0adf60 0x7f965c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965affd700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c06c000 0x7f965c151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 -- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c06c000 msgr2=0x7f965c151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c06c000 0x7f965c151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 -- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c0adf60 msgr2=0x7f965c156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c0adf60 0x7f965c156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 -- 172.21.15.77:0/4048956027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9654009a70 con 0x7f965c06dae0 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965b7fe700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c0adf60 0x7f965c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f965a7fc700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15a720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f964c00c450 tx=0x7f964c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c004c80 con 0x7f965c06dae0 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c004e00 con 0x7f965c06dae0 2024-01-15T14:35:17.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c007310 con 0x7f965c06dae0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c1e4820 con 0x7f965c06dae0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.286+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c158060 con 0x7f965c06dae0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.287+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965c06f7e0 con 0x7f965c06dae0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.288+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f964c007470 con 0x7f965c06dae0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.288+0000 7f9643fff700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f964806b350 0x7f964806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.288+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f964c0896b0 con 0x7f965c06dae0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.288+0000 7f965affd700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f964806b350 0x7f964806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.288+0000 7f965affd700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f964806b350 0x7f964806d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9650000f60 tx=0x7f9650009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.292+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f964c054520 con 0x7f965c06dae0 2024-01-15T14:35:17.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.486+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f965c06def0 con 0x7f965c06dae0 2024-01-15T14:35:17.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.486+0000 7f9643fff700 1 -- 172.21.15.77:0/4048956027 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f965c06def0 con 0x7f965c06dae0 2024-01-15T14:35:17.488 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.488+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f964806b350 msgr2=0x7f964806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.488+0000 7f96619c4700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f964806b350 0x7f964806d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9650000f60 tx=0x7f9650009040 comp rx=0 tx=0).stop 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.488+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 msgr2=0x7f965c15a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.488+0000 7f96619c4700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15a720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f964c00c450 tx=0x7f964c00c810 comp rx=0 tx=0).stop 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 shutdown_connections 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f964806b350 0x7f964806d810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f965c06c000 0x7f965c151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f965c0adf60 0x7f965c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 --2- 172.21.15.77:0/4048956027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f965c06dae0 0x7f965c15a720 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 >> 172.21.15.77:0/4048956027 conn(0x7f965c071bb0 msgr2=0x7f965c06f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:17.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 shutdown_connections 2024-01-15T14:35:17.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.489+0000 7f96619c4700 1 -- 172.21.15.77:0/4048956027 wait complete. 2024-01-15T14:35:17.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:17.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.625+0000 7fe6493cf700 1 Processor -- start 2024-01-15T14:35:17.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.625+0000 7fe6493cf700 1 -- start start 2024-01-15T14:35:17.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6493cf700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe6440d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6493cf700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe6440b98e0 0x7fe6440d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6493cf700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe644162af0 con 0x7fe644161510 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6493cf700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6440d13e0 con 0x7fe6440b98e0 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6493cf700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe644161f70 con 0x7fe6440cf330 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe642ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe644161510 0x7fe6440cec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38788/0 (socket says 172.21.15.77:38788) 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe642ffd700 1 -- 172.21.15.77:0/343532246 learned_addr learned my addr 172.21.15.77:0/343532246 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6427fc700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe6440d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe6437fe700 1 --2- 172.21.15.77:0/343532246 >> v2:172.21.15.159:3300/0 conn(0x7fe6440b98e0 0x7fe6440d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874715176 0 0) 0x7fe644162af0 con 0x7fe644161510 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe62c0035b0 con 0x7fe644161510 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277933130 0 0) 0x7fe62c0035b0 con 0x7fe644161510 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 msgr2=0x7fe6440d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe6440d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 >> v2:172.21.15.159:3300/0 conn(0x7fe6440b98e0 msgr2=0x7fe6440d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 --2- 172.21.15.77:0/343532246 >> v2:172.21.15.159:3300/0 conn(0x7fe6440b98e0 0x7fe6440d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe644162250 con 0x7fe644161510 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.626+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fe638001670 con 0x7fe644161510 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 >> v1:172.21.15.77:6789/0 conn(0x7fe644161510 legacy=0x7fe6440cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe62c003a00 0x7fe62c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe62c004850 0x7fe62c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe62c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440d13e0 con 0x7fe62c004850 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe644161f70 con 0x7fe62c003a00 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c00ba40 con 0x7fe6440cf330 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe62c004850 0x7fe62c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe642ffd700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe62c003a00 0x7fe62c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6437fe700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe62c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 -- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 msgr2=0x7fe62c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe62c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 -- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe62c003a00 msgr2=0x7fe62c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe62c003a00 0x7fe62c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 -- 172.21.15.77:0/343532246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c0035b0 con 0x7fe62c004850 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6437fe700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6440cf330 0x7fe62c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe6427fc700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe62c004850 0x7fe62c004cd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe634009b60 tx=0x7fe62c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2d26b193de1c50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63400e660 con 0x7fe62c004850 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63400e7c0 con 0x7fe62c004850 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.627+0000 7fe641ffb700 1 -- 172.21.15.77:0/343532246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63400eb30 con 0x7fe62c004850 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6493cf700 1 -- 172.21.15.77:0/343532246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe62c004850 msgr2=0x7fe62c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6493cf700 1 --2- 172.21.15.77:0/343532246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe62c004850 0x7fe62c004cd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe634009b60 tx=0x7fe62c005880 comp rx=0 tx=0).stop 2024-01-15T14:35:17.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6437fe700 1 -- 172.21.15.77:0/343532246 reap_dead start 2024-01-15T14:35:17.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6493cf700 1 -- 172.21.15.77:0/343532246 shutdown_connections 2024-01-15T14:35:17.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6493cf700 1 -- 172.21.15.77:0/343532246 >> 172.21.15.77:0/343532246 conn(0x7fe64406b410 msgr2=0x7fe64406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:17.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6493cf700 1 -- 172.21.15.77:0/343532246 shutdown_connections 2024-01-15T14:35:17.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.628+0000 7fe6493cf700 1 -- 172.21.15.77:0/343532246 wait complete. 2024-01-15T14:35:17.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.629+0000 7fe6493cf700 1 Processor -- start 2024-01-15T14:35:17.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.629+0000 7fe6493cf700 1 -- start start 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe62c004850 0x7fe6440b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6441534c0 0x7fe6440b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 0x7fe6441f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441f7890 con 0x7fe6441f6ea0 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440b6730 con 0x7fe6441534c0 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440b5f30 con 0x7fe62c004850 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 0x7fe6441f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 0x7fe6441f7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57908/0 (socket says 172.21.15.77:57908) 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 -- 172.21.15.77:0/3436713065 learned_addr learned my addr 172.21.15.77:0/3436713065 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe642ffd700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe62c004850 0x7fe6440b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6427fc700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6441534c0 0x7fe6440b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 -- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe62c004850 msgr2=0x7fe6440b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe62c004850 0x7fe6440b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 -- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6441534c0 msgr2=0x7fe6440b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6441534c0 0x7fe6440b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 -- 172.21.15.77:0/3436713065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe644153940 con 0x7fe6441f6ea0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6427fc700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6441534c0 0x7fe6440b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6437fe700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 0x7fe6441f7350 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe6440d1630 tx=0x7fe63c0104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00e050 con 0x7fe6441f6ea0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c00f040 con 0x7fe6441f6ea0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6440b6170 con 0x7fe6441f6ea0 2024-01-15T14:35:17.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c010e90 con 0x7fe6441f6ea0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.630+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6440b44f0 con 0x7fe6441f6ea0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.631+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe644068450 con 0x7fe6441f6ea0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.632+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe63c015440 con 0x7fe6441f6ea0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.632+0000 7fe62b7fe700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe63006b350 0x7fe63006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.632+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe63c089dd0 con 0x7fe6441f6ea0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.632+0000 7fe642ffd700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe63006b350 0x7fe63006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.633+0000 7fe642ffd700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe63006b350 0x7fe63006d810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe6440d2fd0 tx=0x7fe638009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.637+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe63c054c30 con 0x7fe6441f6ea0 2024-01-15T14:35:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.828+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe644069020 con 0x7fe6441f6ea0 2024-01-15T14:35:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.829+0000 7fe62b7fe700 1 -- 172.21.15.77:0/3436713065 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe63c058250 con 0x7fe6441f6ea0 2024-01-15T14:35:17.830 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe63006b350 msgr2=0x7fe63006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe63006b350 0x7fe63006d810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe6440d2fd0 tx=0x7fe638009040 comp rx=0 tx=0).stop 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 msgr2=0x7fe6441f7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 0x7fe6441f7350 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe6440d1630 tx=0x7fe63c0104a0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 shutdown_connections 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe63006b350 0x7fe63006d810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe62c004850 0x7fe6440b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6441f6ea0 0x7fe6441f7350 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 --2- 172.21.15.77:0/3436713065 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe6441534c0 0x7fe6440b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 >> 172.21.15.77:0/3436713065 conn(0x7fe64406b410 msgr2=0x7fe6440d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 shutdown_connections 2024-01-15T14:35:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.831+0000 7fe6493cf700 1 -- 172.21.15.77:0/3436713065 wait complete. 2024-01-15T14:35:17.844 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:17.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.964+0000 7f92c97b0700 1 Processor -- start 2024-01-15T14:35:17.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.964+0000 7f92c97b0700 1 -- start start 2024-01-15T14:35:17.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c97b0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 0x7f92c40b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c97b0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40d1190 con 0x7f92c40d0870 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c97b0700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40d20b0 con 0x7f92c415e1e0 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c97b0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40d3150 con 0x7f92c40d0d00 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c27fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f92c40d0870 0x7f92c40b6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38798/0 (socket says 172.21.15.77:38798) 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c27fc700 1 -- 172.21.15.77:0/2480110583 learned_addr learned my addr 172.21.15.77:0/2480110583 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c37fe700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 0x7f92c40b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820543615 0 0) 0x7f92c40d1190 con 0x7f92c40d0870 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac0035b0 con 0x7f92c40d0870 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611231408 0 0) 0x7f92ac0035b0 con 0x7f92c40d0870 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 msgr2=0x7f92c40b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 0x7f92c40b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 >> v1:172.21.15.159:6789/0 conn(0x7f92c415e1e0 legacy=0x7f92c40d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40d2390 con 0x7f92c40d0870 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f92b8001670 con 0x7f92c40d0870 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 >> v1:172.21.15.77:6789/0 conn(0x7f92c40d0870 legacy=0x7f92c40b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92ac003a30 0x7f92ac003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 0x7f92ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40d3150 con 0x7f92ac004850 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac00bb50 con 0x7f92ac003a30 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac00bd10 con 0x7f92c40d0d00 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c27fc700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c2ffd700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92ac003a30 0x7f92ac003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.965+0000 7f92c37fe700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 0x7f92ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c27fc700 1 -- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 msgr2=0x7f92ac005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c27fc700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c40d0d00 0x7f92ac005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c27fc700 1 -- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92ac003a30 msgr2=0x7f92ac003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c27fc700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92ac003a30 0x7f92ac003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c27fc700 1 -- 172.21.15.77:0/2480110583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac0035b0 con 0x7f92ac004850 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c27fc700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92ac004cd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c210 tx=0x7f92ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3828ed4bad4338e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8007c10 con 0x7f92ac004850 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b800cdd0 con 0x7f92ac004850 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c1ffb700 1 -- 172.21.15.77:0/2480110583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8005ba0 con 0x7f92ac004850 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c97b0700 1 -- 172.21.15.77:0/2480110583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 msgr2=0x7f92ac004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c97b0700 1 --2- 172.21.15.77:0/2480110583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92ac004cd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c210 tx=0x7f92ac005880 comp rx=0 tx=0).stop 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c37fe700 1 -- 172.21.15.77:0/2480110583 reap_dead start 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c97b0700 1 -- 172.21.15.77:0/2480110583 shutdown_connections 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c97b0700 1 -- 172.21.15.77:0/2480110583 >> 172.21.15.77:0/2480110583 conn(0x7f92c40b5b50 msgr2=0x7f92c40b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.966+0000 7f92c97b0700 1 -- 172.21.15.77:0/2480110583 shutdown_connections 2024-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.967+0000 7f92c97b0700 1 -- 172.21.15.77:0/2480110583 wait complete. 2024-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.968+0000 7f92c97b0700 1 Processor -- start 2024-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.968+0000 7f92c97b0700 1 -- start start 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c97b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92c4151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c97b0700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c41612d0 0x7f92c40cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c97b0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4154890 0x7f92c40ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c97b0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c415f480 con 0x7f92ac004850 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c97b0700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c4151aa0 con 0x7f92c41612d0 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c97b0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c415e870 con 0x7f92c4154890 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92c4151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92c4151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57916/0 (socket says 172.21.15.77:57916) 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c2ffd700 1 -- 172.21.15.77:0/3893273754 learned_addr learned my addr 172.21.15.77:0/3893273754 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c27fc700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c41612d0 0x7f92c40cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c37fe700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4154890 0x7f92c40ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c2ffd700 1 -- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4154890 msgr2=0x7f92c40ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.969+0000 7f92c2ffd700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4154890 0x7f92c40ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c2ffd700 1 -- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c41612d0 msgr2=0x7f92c40cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c2ffd700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c41612d0 0x7f92c40cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c2ffd700 1 -- 172.21.15.77:0/3893273754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c415eb50 con 0x7f92ac004850 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c37fe700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4154890 0x7f92c40ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c2ffd700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92c4151560 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f92b400f7f0 tx=0x7f92b400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4005320 con 0x7f92ac004850 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b4005480 con 0x7f92ac004850 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c415ee30 con 0x7f92ac004850 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4006aa0 con 0x7f92ac004850 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.970+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c40cbe80 con 0x7f92ac004850 2024-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.971+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c40751e0 con 0x7f92ac004850 2024-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.971+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f92b4006c00 con 0x7f92ac004850 2024-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.972+0000 7f92ab7fe700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b350 0x7f92b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.972+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92b4089100 con 0x7f92ac004850 2024-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.972+0000 7f92c27fc700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b350 0x7f92b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.972+0000 7f92c27fc700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b350 0x7f92b006d810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f92b8015040 tx=0x7f92b8007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:17.976+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92b4053f60 con 0x7f92ac004850 2024-01-15T14:35:18.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.209+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92c40b92b0 con 0x7f92ac004850 2024-01-15T14:35:18.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.210+0000 7f92ab7fe700 1 -- 172.21.15.77:0/3893273754 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f92b4057580 con 0x7f92ac004850 2024-01-15T14:35:18.211 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:18.211 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b350 msgr2=0x7f92b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b350 0x7f92b006d810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f92b8015040 tx=0x7f92b8007000 comp rx=0 tx=0).stop 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 msgr2=0x7f92c4151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92c4151560 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f92b400f7f0 tx=0x7f92b400fbb0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 shutdown_connections 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b350 0x7f92b006d810 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4154890 0x7f92c40ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92ac004850 0x7f92c4151560 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 --2- 172.21.15.77:0/3893273754 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c41612d0 0x7f92c40cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 >> 172.21.15.77:0/3893273754 conn(0x7f92c40b5b50 msgr2=0x7f92c4153330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:18.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 shutdown_connections 2024-01-15T14:35:18.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.212+0000 7f92c97b0700 1 -- 172.21.15.77:0/3893273754 wait complete. 2024-01-15T14:35:18.215 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:18.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:18.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.346+0000 7fc3f28c2700 1 Processor -- start 2024-01-15T14:35:18.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.346+0000 7fc3f28c2700 1 -- start start 2024-01-15T14:35:18.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3f28c2700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc3ec0d22f0 0x7fc3ec0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3f28c2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec0d0150 con 0x7fc3ec15e080 2024-01-15T14:35:18.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3f28c2700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec0cee50 con 0x7fc3ec0ce4d0 2024-01-15T14:35:18.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3f28c2700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec15e460 con 0x7fc3ec0d22f0 2024-01-15T14:35:18.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3ebfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc3ec15e080 0x7fc3ec0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38800/0 (socket says 172.21.15.77:38800) 2024-01-15T14:35:18.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3ebfff700 1 -- 172.21.15.77:0/1903687984 learned_addr learned my addr 172.21.15.77:0/1903687984 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376937353 0 0) 0x7fc3ec0d0150 con 0x7fc3ec15e080 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d40035b0 con 0x7fc3ec15e080 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3f0e5f700 1 --2- 172.21.15.77:0/1903687984 >> v2:172.21.15.159:3300/0 conn(0x7fc3ec0d22f0 0x7fc3ec0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469937459 0 0) 0x7fc3d40035b0 con 0x7fc3ec15e080 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 >> v2:172.21.15.159:3300/0 conn(0x7fc3ec0d22f0 msgr2=0x7fc3ec0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 --2- 172.21.15.77:0/1903687984 >> v2:172.21.15.159:3300/0 conn(0x7fc3ec0d22f0 0x7fc3ec0d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 >> v1:172.21.15.159:6789/0 conn(0x7fc3ec0ce4d0 legacy=0x7fc3ec0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec0cf130 con 0x7fc3ec15e080 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fc3e0001670 con 0x7fc3ec15e080 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3f0e5f700 1 --2- 172.21.15.77:0/1903687984 >> v2:172.21.15.159:3300/0 conn(0x7fc3ec0d22f0 0x7fc3ec0d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 >> v1:172.21.15.77:6789/0 conn(0x7fc3ec15e080 legacy=0x7fc3ec0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3d4005210 0x7fc3d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec15e460 con 0x7fc3d4004850 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400c160 con 0x7fc3d4003a00 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.347+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400c320 con 0x7fc3d4005210 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3ebfff700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3f0e5f700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3d4005210 0x7fc3d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3d4005210 msgr2=0x7fc3d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3d4005210 0x7fc3d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4003a00 msgr2=0x7fc3d4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4003a00 0x7fc3d4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/1903687984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40035b0 con 0x7fc3d4004850 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3f0e5f700 1 -- 172.21.15.77:0/1903687984 reap_dead start 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0017c0 tx=0x7fc3d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7a11063002c583 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0052d0 con 0x7fc3d4004850 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc005430 con 0x7fc3d4004850 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3eaffd700 1 -- 172.21.15.77:0/1903687984 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0038c0 con 0x7fc3d4004850 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3f28c2700 1 -- 172.21.15.77:0/1903687984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3d4004850 msgr2=0x7fc3d4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.348+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0017c0 tx=0x7fc3d4005880 comp rx=0 tx=0).stop 2024-01-15T14:35:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.349+0000 7fc3f28c2700 1 -- 172.21.15.77:0/1903687984 shutdown_connections 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.349+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/1903687984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3d4004850 0x7fc3d4004cd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.349+0000 7fc3f28c2700 1 -- 172.21.15.77:0/1903687984 >> 172.21.15.77:0/1903687984 conn(0x7fc3ec073130 msgr2=0x7fc3ec09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.349+0000 7fc3f28c2700 1 -- 172.21.15.77:0/1903687984 shutdown_connections 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.349+0000 7fc3f28c2700 1 -- 172.21.15.77:0/1903687984 wait complete. 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.350+0000 7fc3f28c2700 1 Processor -- start 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.350+0000 7fc3f28c2700 1 -- start start 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f28c2700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4004850 0x7fc3ec09ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f28c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 0x7fc3ec1e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f28c2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3ec1e84c0 0x7fc3ec09c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f28c2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1e8d40 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f28c2700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1e9020 con 0x7fc3d4004850 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f28c2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1e9300 con 0x7fc3ec1e84c0 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3eb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 0x7fc3ec1e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3eb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 0x7fc3ec1e7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57932/0 (socket says 172.21.15.77:57932) 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/2523408020 learned_addr learned my addr 172.21.15.77:0/2523408020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3f0e5f700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3ec1e84c0 0x7fc3ec09c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.351+0000 7fc3ebfff700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4004850 0x7fc3ec09ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3ec1e84c0 msgr2=0x7fc3ec09c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3ec1e84c0 0x7fc3ec09c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4004850 msgr2=0x7fc3ec09ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4004850 0x7fc3ec09ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3eb7fe700 1 -- 172.21.15.77:0/2523408020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec1e95e0 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3ebfff700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4004850 0x7fc3ec09ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.352+0000 7fc3eb7fe700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 0x7fc3ec1e7f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc015040 tx=0x7fc3dc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.353+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc001290 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.353+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc01d480 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.353+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc01d830 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.353+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec1e98c0 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.353+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec09f490 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.353+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec067f60 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.354+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc3dc019040 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.354+0000 7fc3e8ff9700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc3d806b350 0x7fc3d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.354+0000 7fc3ebfff700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc3d806b350 0x7fc3d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.354+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3dc08a5f0 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.355+0000 7fc3ebfff700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc3d806b350 0x7fc3d806d810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0007dd0 tx=0x7fc3e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:18.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.359+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3dc055610 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.550+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3ec0b7bd0 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.552+0000 7fc3e8ff9700 1 -- 172.21.15.77:0/2523408020 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fc3dc02d090 con 0x7fc3ec1e9dc0 2024-01-15T14:35:18.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:18.553 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc3d806b350 msgr2=0x7fc3d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc3d806b350 0x7fc3d806d810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0007dd0 tx=0x7fc3e0009040 comp rx=0 tx=0).stop 2024-01-15T14:35:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 msgr2=0x7fc3ec1e7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 0x7fc3ec1e7f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc015040 tx=0x7fc3dc001120 comp rx=0 tx=0).stop 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 shutdown_connections 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc3d806b350 0x7fc3d806d810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc3ec1e84c0 0x7fc3ec09c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc3d4004850 0x7fc3ec09ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 --2- 172.21.15.77:0/2523408020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3ec1e9dc0 0x7fc3ec1e7f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 >> 172.21.15.77:0/2523408020 conn(0x7fc3ec073130 msgr2=0x7fc3ec15ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 shutdown_connections 2024-01-15T14:35:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.554+0000 7fc3f28c2700 1 -- 172.21.15.77:0/2523408020 wait complete. 2024-01-15T14:35:18.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:18.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.688+0000 7f299e037700 1 Processor -- start 2024-01-15T14:35:18.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.688+0000 7f299e037700 1 -- start start 2024-01-15T14:35:18.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f299e037700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f299806ee50 0x7f2998074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f299e037700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998074e90 con 0x7f299815c140 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f299e037700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29980748b0 con 0x7f2998073f90 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f299e037700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299815c700 con 0x7f299806ee50 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f2997fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f299815c140 0x7f29980cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38810/0 (socket says 172.21.15.77:38810) 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f2997fff700 1 -- 172.21.15.77:0/1141448990 learned_addr learned my addr 172.21.15.77:0/1141448990 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.689+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122809173 0 0) 0x7f2998074e90 con 0x7f299815c140 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29800035b0 con 0x7f299815c140 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1141448990 >> v2:172.21.15.159:3300/0 conn(0x7f299806ee50 0x7f2998074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 857243330 0 0) 0x7f29800035b0 con 0x7f299815c140 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 >> v2:172.21.15.159:3300/0 conn(0x7f299806ee50 msgr2=0x7f2998074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:18.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 --2- 172.21.15.77:0/1141448990 >> v2:172.21.15.159:3300/0 conn(0x7f299806ee50 0x7f2998074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 >> v1:172.21.15.159:6789/0 conn(0x7f2998073f90 legacy=0x7f29980ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299815c9e0 con 0x7f299815c140 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f29900019f0 con 0x7f299815c140 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 >> v1:172.21.15.77:6789/0 conn(0x7f299815c140 legacy=0x7f29980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980003a00 0x7f2980003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2980004850 0x7f2980004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2980005260 0x7f298000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299815c700 con 0x7f2980004850 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298000c190 con 0x7f2980003a00 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298000c350 con 0x7f2980005260 2024-01-15T14:35:18.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2980004850 0x7f2980004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29977fe700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980003a00 0x7f2980003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2997fff700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2980005260 0x7f298000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 -- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2980005260 msgr2=0x7f298000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2980005260 0x7f298000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 -- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980003a00 msgr2=0x7f2980003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980003a00 0x7f2980003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 -- 172.21.15.77:0/1141448990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29800035b0 con 0x7f2980004850 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2997fff700 1 -- 172.21.15.77:0/1141448990 reap_dead start 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2980004850 0x7f2980004cd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f298c009b60 tx=0x7f2980005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90af4bf0068a2354 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00e6c0 con 0x7f2980004850 2024-01-15T14:35:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c00e820 con 0x7f2980004850 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.690+0000 7f29967fc700 1 -- 172.21.15.77:0/1141448990 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00eb90 con 0x7f2980004850 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 -- 172.21.15.77:0/1141448990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2980004850 msgr2=0x7f2980004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2980004850 0x7f2980004cd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f298c009b60 tx=0x7f2980005880 comp rx=0 tx=0).stop 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 -- 172.21.15.77:0/1141448990 shutdown_connections 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 --2- 172.21.15.77:0/1141448990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2980004850 0x7f2980004cd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 -- 172.21.15.77:0/1141448990 >> 172.21.15.77:0/1141448990 conn(0x7f29980cd570 msgr2=0x7f29980cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 -- 172.21.15.77:0/1141448990 shutdown_connections 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.691+0000 7f299e037700 1 -- 172.21.15.77:0/1141448990 wait complete. 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.692+0000 7f299e037700 1 Processor -- start 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.692+0000 7f299e037700 1 -- start start 2024-01-15T14:35:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f299e037700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980004850 0x7f299813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f299e037700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 0x7f2998151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f299e037700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f299814bfe0 0x7f299814c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f299e037700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299814f2e0 con 0x7f2998150fc0 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f299e037700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2998151980 con 0x7f2980004850 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f299e037700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299813da20 con 0x7f299814bfe0 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 0x7f2998151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 0x7f2998151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57954/0 (socket says 172.21.15.77:57954) 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 -- 172.21.15.77:0/1221720644 learned_addr learned my addr 172.21.15.77:0/1221720644 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f29977fe700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980004850 0x7f299813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2997fff700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f299814bfe0 0x7f299814c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 -- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f299814bfe0 msgr2=0x7f299814c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f299814bfe0 0x7f299814c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 -- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980004850 msgr2=0x7f299813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980004850 0x7f299813d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.693+0000 7f2996ffd700 1 -- 172.21.15.77:0/1221720644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299814cdb0 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.694+0000 7f2996ffd700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 0x7f2998151440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f298000c160 tx=0x7f298c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.694+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00e6c0 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.694+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c0038c0 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.694+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c003c70 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.694+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299814e940 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.694+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299814ef00 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.695+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2998067f60 con 0x7f2998150fc0 2024-01-15T14:35:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.696+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f298c013470 con 0x7f2998150fc0 2024-01-15T14:35:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.696+0000 7f297ffff700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f298406b350 0x7f298406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.696+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f298c08b2d0 con 0x7f2998150fc0 2024-01-15T14:35:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.696+0000 7f29977fe700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f298406b350 0x7f298406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.696+0000 7f29977fe700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f298406b350 0x7f298406d810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2988000c00 tx=0x7f2988009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.701+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f298c0562f0 con 0x7f2998150fc0 2024-01-15T14:35:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.933+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29980d0860 con 0x7f2998150fc0 2024-01-15T14:35:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.933+0000 7f297ffff700 1 -- 172.21.15.77:0/1221720644 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f298c059910 con 0x7f2998150fc0 2024-01-15T14:35:18.934 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:18.934 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:18.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f298406b350 msgr2=0x7f298406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f298406b350 0x7f298406d810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2988000c00 tx=0x7f2988009040 comp rx=0 tx=0).stop 2024-01-15T14:35:18.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 msgr2=0x7f2998151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:18.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 0x7f2998151440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f298000c160 tx=0x7f298c0047c0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 shutdown_connections 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f298406b350 0x7f298406d810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f299814bfe0 0x7f299814c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2980004850 0x7f299813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 --2- 172.21.15.77:0/1221720644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2998150fc0 0x7f2998151440 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 >> 172.21.15.77:0/1221720644 conn(0x7f29980cd570 msgr2=0x7f2998156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 shutdown_connections 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:18.935+0000 7f299e037700 1 -- 172.21.15.77:0/1221720644 wait complete. 2024-01-15T14:35:18.938 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:19.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T14:35:04.589892+0000', 'modified': '2024-01-15T14:35:07.623076+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4278}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4278': {'gid': 4278, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.159:6835/1253210024', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6833', 'nonce': 1253210024}, {'type': 'v1', 'addr': '172.21.15.159:6835', 'nonce': 1253210024}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T14:35:19.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T14:35:19.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-15T14:35:19.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.475+0000 7fc0bc88b700 1 Processor -- start 2024-01-15T14:35:19.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.475+0000 7fc0bc88b700 1 -- start start 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.475+0000 7fc0bc88b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0bc88b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0b81603c0 0x7fc0b80d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0bc88b700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80d0770 con 0x7fc0b815bed0 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0bc88b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b815c2b0 con 0x7fc0b81603c0 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0bc88b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80cd100 con 0x7fc0b80d2160 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0b81603c0 0x7fc0b80d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0b81603c0 0x7fc0b80d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40604/0 (socket says 172.21.15.77:40604) 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3604369698 learned_addr learned my addr 172.21.15.77:0/3604369698 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/3604369698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3604369698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 msgr2=0x7fc0b80b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3604369698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3604369698 >> v1:172.21.15.159:6789/0 conn(0x7fc0b815bed0 legacy=0x7fc0b80d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3604369698 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80d09a0 con 0x7fc0b81603c0 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3604369698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b80d0230 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009e50 tx=0x7fc0b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08db5f68d167f0b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b559a700 1 -- 172.21.15.77:0/3604369698 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000e070 con 0x7fc0b81603c0 2024-01-15T14:35:19.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b559a700 1 -- 172.21.15.77:0/3604369698 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b000b910 con 0x7fc0b81603c0 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.476+0000 7fc0b559a700 1 -- 172.21.15.77:0/3604369698 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000bcc0 con 0x7fc0b81603c0 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 -- 172.21.15.77:0/3604369698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 msgr2=0x7fc0b80d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/3604369698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b80d0230 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009e50 tx=0x7fc0b000b330 comp rx=0 tx=0).stop 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 -- 172.21.15.77:0/3604369698 shutdown_connections 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/3604369698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b80d0230 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/3604369698 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 -- 172.21.15.77:0/3604369698 >> 172.21.15.77:0/3604369698 conn(0x7fc0b8073130 msgr2=0x7fc0b809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 -- 172.21.15.77:0/3604369698 shutdown_connections 2024-01-15T14:35:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.477+0000 7fc0bc88b700 1 -- 172.21.15.77:0/3604369698 wait complete. 2024-01-15T14:35:19.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.478+0000 7fc0bc88b700 1 Processor -- start 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.478+0000 7fc0bc88b700 1 -- start start 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0bc88b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80c4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0bc88b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b815bed0 0x7fc0b815c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0bc88b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0bc88b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b806b470 con 0x7fc0b81603c0 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0bc88b700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b806b6c0 con 0x7fc0b815bed0 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0bc88b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b806b9a0 con 0x7fc0b80d2160 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b815c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40606/0 (socket says 172.21.15.77:40606) 2024-01-15T14:35:19.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/2029121491 learned_addr learned my addr 172.21.15.77:0/2029121491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b659c700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b815bed0 0x7fc0b815c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 msgr2=0x7fc0b80c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b815bed0 msgr2=0x7fc0b815c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.479+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b815bed0 0x7fc0b815c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/2029121491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0009b30 con 0x7fc0b81603c0 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b815c960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80cf7d0 tx=0x7fc0b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0a6ffd700 1 -- 172.21.15.77:0/2029121491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0004960 con 0x7fc0b81603c0 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0a6ffd700 1 -- 172.21.15.77:0/2029121491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b00099b0 con 0x7fc0b81603c0 2024-01-15T14:35:19.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b806bc80 con 0x7fc0b81603c0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0a6ffd700 1 -- 172.21.15.77:0/2029121491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b001c680 con 0x7fc0b81603c0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.480+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b806c240 con 0x7fc0b81603c0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.481+0000 7fc0a6ffd700 1 -- 172.21.15.77:0/2029121491 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc0b001c7e0 con 0x7fc0b81603c0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.482+0000 7fc0a6ffd700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc09806b350 0x7fc09806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.482+0000 7fc0b659c700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc09806b350 0x7fc09806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.482+0000 7fc0a6ffd700 1 -- 172.21.15.77:0/2029121491 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0b008b880 con 0x7fc0b81603c0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.482+0000 7fc0b659c700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc09806b350 0x7fc09806d810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac000f60 tx=0x7fc0ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.483+0000 7fc0c8a87b80 1 Processor -- start 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 -- start start 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269f33cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f10190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269eed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269e60480 con 0x55a269eeadd0 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269e46770 con 0x55a269f15410 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.484+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a269e44090 con 0x55a269e78ce0 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269eed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269eed1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40610/0 (socket says 172.21.15.77:40610) 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/1625118855 learned_addr learned my addr 172.21.15.77:0/1625118855 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b659c700 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269f33cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f10190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 msgr2=0x55a269f33cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269f33cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 msgr2=0x55a269f10190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f10190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/1625118855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a269c68b10 con 0x55a269eeadd0 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269eed1d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0089e60 tx=0x7fc0b0059ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5535c28a241837e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc097fff700 1 -- 172.21.15.77:0/1625118855 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0062740 con 0x55a269eeadd0 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc097fff700 1 -- 172.21.15.77:0/1625118855 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b00092e0 con 0x55a269eeadd0 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.485+0000 7fc097fff700 1 -- 172.21.15.77:0/1625118855 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00587e0 con 0x55a269eeadd0 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/1625118855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 msgr2=0x55a269eed1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269eed1d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0089e60 tx=0x7fc0b0059ea0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/1625118855 shutdown_connections 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269f33cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269eed1d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/1625118855 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f10190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/1625118855 >> 172.21.15.77:0/1625118855 conn(0x55a269e1a410 msgr2=0x55a269f32830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/1625118855 shutdown_connections 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/1625118855 wait complete. 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 -- start start 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.486+0000 7fc0c8a87b80 1 Processor -- start 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269f1b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f44dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269f15410 0x55a269eebed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269e3f980 con 0x55a269eeadd0 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269ddf000 con 0x55a269e78ce0 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a269ddeca0 con 0x55a269f15410 2024-01-15T14:35:19.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0c8a87b80 10 client.0 _ll_register_callbacks cb 0x55a269c42350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0b5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f44dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0b5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f44dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40620/0 (socket says 172.21.15.77:40620) 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.487+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/3277678189 learned_addr learned my addr 172.21.15.77:0/3277678189 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b659c700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269f1b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269f15410 0x55a269eebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269f15410 msgr2=0x55a269eebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269f15410 0x55a269eebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 msgr2=0x55a269f1b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269f1b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/3277678189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b806c240 con 0x55a269eeadd0 2024-01-15T14:35:19.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269f15410 0x55a269eebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b659c700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269f1b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f44dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009bd0 tx=0x7fc0a8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e8971d82902341 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0957fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0957fa700 1 -- 172.21.15.77:0/3277678189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8013bb0 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0957fa700 1 -- 172.21.15.77:0/3277678189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0a8013d10 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a269e5f450 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0957fa700 1 -- 172.21.15.77:0/3277678189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a8013ed0 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 10 client.4638 fetch_fsmap learned FSMap version 8 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55a269e01290 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0957fa700 1 -- 172.21.15.77:0/3277678189 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a269e5f450 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0957fa700 1 -- 172.21.15.77:0/3277678189 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc0a8005a20 con 0x55a269eeadd0 2024-01-15T14:35:19.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 10 client.4638 fetch_fsmap finished waiting for FSMap version 8 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 10 client.4638 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 20 client.4638 populate_metadata read hostname 'smithi077' 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f0f160 0x55a269f33cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 4 client.4638 mds_command: new command op to 4278 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.488+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 --> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a269e38960 con 0x55a269f0f160 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.489+0000 7fc0b659c700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f0f160 0x55a269f33cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.489+0000 7fc0957fa700 10 client.4638 ms_handle_connect on v2:172.21.15.159:6833/1253210024 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.489+0000 7fc0b659c700 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f0f160 0x55a269f33cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2f98255e9e9393 server_cookie=56ee3b79094d7bbe in_seq=0 out_seq=0 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0957fa700 1 -- 172.21.15.77:0/3277678189 <== mds.0 v2:172.21.15.159:6833/1253210024 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a269e38960 con 0x55a269f0f160 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0957fa700 10 client.4638 handle_command_reply: tid=0 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 client.4638 shutdown 2024-01-15T14:35:19.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 msgr2=0x55a269f44dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f44dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009bd0 tx=0x7fc0a8009f90 comp rx=0 tx=0).stop 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 shutdown_connections 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f0f160 0x55a269f33cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269f15410 0x55a269eebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269f1b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3277678189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f44dd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0957fa700 0 client.4638 ms_handle_reset on v2:172.21.15.159:6833/1253210024 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 >> 172.21.15.77:0/3277678189 conn(0x55a269e1a410 msgr2=0x55a269f17a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 shutdown_connections 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3277678189 wait complete. 2024-01-15T14:35:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.490+0000 7fc0c8a87b80 20 client.4638 trim_cache size 0 max 16384 2024-01-15T14:35:19.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 Processor -- start 2024-01-15T14:35:19.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 -- start start 2024-01-15T14:35:19.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269ee8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269eeadd0 0x55a269f4b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 0x55a269f17a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269cbe650 con 0x55a269f15410 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269ee8250 con 0x55a269e78ce0 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a269f0f750 con 0x55a269eeadd0 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 0x55a269f17a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 0x55a269f17a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40622/0 (socket says 172.21.15.77:40622) 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3735160128 learned_addr learned my addr 172.21.15.77:0/3735160128 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269eeadd0 0x55a269f4b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.509+0000 7fc0b659c700 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269ee8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269eeadd0 msgr2=0x55a269f4b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269eeadd0 0x55a269f4b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 msgr2=0x55a269ee8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269ee8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0b6d9d700 1 -- 172.21.15.77:0/3735160128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a269cc0c80 con 0x55a269f15410 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 0x55a269f17a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0056940 tx=0x7fc0b006f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9b8815f199ec83 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0a4ff9700 1 -- 172.21.15.77:0/3735160128 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0062740 con 0x55a269f15410 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0a4ff9700 1 -- 172.21.15.77:0/3735160128 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b0058580 con 0x55a269f15410 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0a4ff9700 1 -- 172.21.15.77:0/3735160128 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0023360 con 0x55a269f15410 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3735160128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 msgr2=0x55a269f17a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 0x55a269f17a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0056940 tx=0x7fc0b006f000 comp rx=0 tx=0).stop 2024-01-15T14:35:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3735160128 shutdown_connections 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269f15410 0x55a269f17a10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269e78ce0 0x55a269ee8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/3735160128 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269eeadd0 0x55a269f4b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.510+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3735160128 >> 172.21.15.77:0/3735160128 conn(0x55a269e1a410 msgr2=0x55a269f2a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.511+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3735160128 shutdown_connections 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.511+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/3735160128 wait complete. 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.511+0000 7fc0c8a87b80 1 -- start start 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.511+0000 7fc0c8a87b80 1 Processor -- start 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269eeb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f145b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269f0b810 con 0x55a269eeadd0 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a269f179f0 con 0x55a269f15410 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a269f17c90 con 0x55a269e78ce0 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0c8a87b80 10 client.0 _ll_register_callbacks cb 0x7fc0a8009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f145b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f145b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40636/0 (socket says 172.21.15.77:40636) 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/4080686416 learned_addr learned my addr 172.21.15.77:0/4080686416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b659c700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269eeb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b6d9d700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 msgr2=0x55a269eeb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269eeb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 msgr2=0x55a269f435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.512+0000 7fc0b5d9b700 1 -- 172.21.15.77:0/4080686416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a269c68b10 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0b5d9b700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f145b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55a269f44d30 tx=0x7fc0a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57268631e7f6bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0957fa700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0957fa700 1 -- 172.21.15.77:0/4080686416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8005ba0 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0957fa700 1 -- 172.21.15.77:0/4080686416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0a8000e40 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0b00092e0 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0957fa700 1 -- 172.21.15.77:0/4080686416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a8004250 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 10 client.4644 fetch_fsmap learned FSMap version 8 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55a269c66fe0 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0957fa700 1 -- 172.21.15.77:0/4080686416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b00092e0 con 0x55a269eeadd0 2024-01-15T14:35:19.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0957fa700 1 -- 172.21.15.77:0/4080686416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc0a80043b0 con 0x55a269eeadd0 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 10 client.4644 fetch_fsmap finished waiting for FSMap version 8 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 10 client.4644 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 20 client.4644 populate_metadata read hostname 'smithi077' 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f07610 0x55a269f079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 4 client.4644 mds_command: new command op to 4278 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 --> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a269e38680 con 0x55a269f07610 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.513+0000 7fc0b659c700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f07610 0x55a269f079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0957fa700 10 client.4644 ms_handle_connect on v2:172.21.15.159:6833/1253210024 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0b659c700 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f07610 0x55a269f079f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dffd158741423b0 server_cookie=268379dae95cf51c in_seq=0 out_seq=0 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0957fa700 1 -- 172.21.15.77:0/4080686416 <== mds.0 v2:172.21.15.159:6833/1253210024 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55a269e38680 con 0x55a269f07610 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0957fa700 10 client.4644 handle_command_reply: tid=0 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 client.4644 shutdown 2024-01-15T14:35:19.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 msgr2=0x55a269f145b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f145b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55a269f44d30 tx=0x7fc0a8004750 comp rx=0 tx=0).stop 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 shutdown_connections 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:6833/1253210024,v1:172.21.15.159:6835/1253210024] conn(0x55a269f07610 0x55a269f079f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x55a269e78ce0 0x55a269eeb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x55a269f15410 0x55a269f435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 --2- 172.21.15.77:0/4080686416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a269eeadd0 0x55a269f145b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0957fa700 0 client.4644 ms_handle_reset on v2:172.21.15.159:6833/1253210024 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 >> 172.21.15.77:0/4080686416 conn(0x55a269e1a410 msgr2=0x55a269f54510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 shutdown_connections 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 1 -- 172.21.15.77:0/4080686416 wait complete. 2024-01-15T14:35:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.514+0000 7fc0c8a87b80 20 client.4644 trim_cache size 0 max 16384 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "a5eae108-559b-4366-a994-4d2898914163", 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4278, 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-01-15T14:35:19.525 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 14.300492716999999, 2024-01-15T14:35:19.526 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 8, 2024-01-15T14:35:19.526 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 21, 2024-01-15T14:35:19.526 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 21, 2024-01-15T14:35:19.526 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 21.843299910999999 2024-01-15T14:35:19.526 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:35:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc09806b350 msgr2=0x7fc09806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc09806b350 0x7fc09806d810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac000f60 tx=0x7fc0ac009040 comp rx=0 tx=0).stop 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 msgr2=0x7fc0b815c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b815c960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80cf7d0 tx=0x7fc0b0004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 shutdown_connections 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc09806b350 0x7fc09806d810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc0b80d2160 0x7fc0b80c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0b81603c0 0x7fc0b815c960 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 --2- 172.21.15.77:0/2029121491 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc0b815bed0 0x7fc0b815c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 >> 172.21.15.77:0/2029121491 conn(0x7fc0b8073130 msgr2=0x7fc0b815f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 shutdown_connections 2024-01-15T14:35:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:19.515+0000 7fc0bc88b700 1 -- 172.21.15.77:0/2029121491 wait complete. 2024-01-15T14:35:19.530 INFO:teuthology.run_tasks:Running task kclient... 2024-01-15T14:35:19.555 INFO:tasks.kclient:Mounting kernel clients... 2024-01-15T14:35:19.555 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-15T14:35:19.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.555 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-15T14:35:19.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.574 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-01-15T14:35:19.643 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-01-15T14:35:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:35:19.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.644 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-15T14:35:19.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.702 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-01-15T14:35:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-01-15T14:35:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:35:19.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.773 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-01-15T14:35:19.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.795 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-01-15T14:35:19.870 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-01-15T14:35:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:35:19.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.872 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-01-15T14:35:19.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:19.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link delete ceph-brx 2024-01-15T14:35:20.003 INFO:teuthology.orchestra.run.smithi159.stderr:Cannot find device "ceph-brx" 2024-01-15T14:35:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T14:35:20.005 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-01-15T14:35:20.005 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T14:35:20.005 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T14:35:20.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T14:35:20.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.127+0000 7f3d225dd700 1 Processor -- start 2024-01-15T14:35:20.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.127+0000 7f3d225dd700 1 -- start start 2024-01-15T14:35:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d225dd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c15af10 0x7f3d1c15e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d225dd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d1c15f630 0x7f3d1c06b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d225dd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c1414c0 con 0x7f3d1c0ce140 2024-01-15T14:35:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d225dd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c15c300 con 0x7f3d1c15f630 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d225dd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0ce9f0 con 0x7f3d1c15af10 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d20b7a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d1c15f630 0x7f3d1c06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d1b7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3d1c0ce140 0x7f3d1c15ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34922/0 (socket says 172.21.15.77:34922) 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d1b7fe700 1 -- 172.21.15.77:0/49558240 learned_addr learned my addr 172.21.15.77:0/49558240 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d1bfff700 1 --2- 172.21.15.77:0/49558240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c15af10 0x7f3d1c15e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d20b7a700 1 -- 172.21.15.77:0/49558240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c15af10 msgr2=0x7f3d1c15e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d20b7a700 1 --2- 172.21.15.77:0/49558240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c15af10 0x7f3d1c15e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d20b7a700 1 -- 172.21.15.77:0/49558240 >> v1:172.21.15.77:6789/0 conn(0x7f3d1c0ce140 legacy=0x7f3d1c15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d1affd700 1 -- 172.21.15.77:0/49558240 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404780981 0 0) 0x7f3d1c1414c0 con 0x7f3d1c0ce140 2024-01-15T14:35:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.128+0000 7f3d20b7a700 1 -- 172.21.15.77:0/49558240 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0cecd0 con 0x7f3d1c15f630 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d20b7a700 1 --2- 172.21.15.77:0/49558240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15f630 0x7f3d1c06b640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009d90 tx=0x7f3d1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6e2898a0a0d6ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d1affd700 1 -- 172.21.15.77:0/49558240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1400e070 con 0x7f3d1c15f630 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d1affd700 1 -- 172.21.15.77:0/49558240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1400bbd0 con 0x7f3d1c15f630 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d1affd700 1 -- 172.21.15.77:0/49558240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d14012430 con 0x7f3d1c15f630 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 -- 172.21.15.77:0/49558240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15f630 msgr2=0x7f3d1c06b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 --2- 172.21.15.77:0/49558240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15f630 0x7f3d1c06b640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009d90 tx=0x7f3d1400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 -- 172.21.15.77:0/49558240 shutdown_connections 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 --2- 172.21.15.77:0/49558240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15f630 0x7f3d1c06b640 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 --2- 172.21.15.77:0/49558240 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c15af10 0x7f3d1c15e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 -- 172.21.15.77:0/49558240 >> 172.21.15.77:0/49558240 conn(0x7f3d1c0cd570 msgr2=0x7f3d1c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:20.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 -- 172.21.15.77:0/49558240 shutdown_connections 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.129+0000 7f3d225dd700 1 -- 172.21.15.77:0/49558240 wait complete. 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.130+0000 7f3d225dd700 1 Processor -- start 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 -- start start 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d1c0ce140 0x7f3d1c1f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 0x7f3d1c13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c13eac0 0x7f3d1c0b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c158280 con 0x7f3d1c15af10 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c13fb80 con 0x7f3d1c0ce140 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d225dd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c13f000 con 0x7f3d1c13eac0 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d1b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 0x7f3d1c13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:20.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d1b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 0x7f3d1c13e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40654/0 (socket says 172.21.15.77:40654) 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.131+0000 7f3d1b7fe700 1 -- 172.21.15.77:0/2318744613 learned_addr learned my addr 172.21.15.77:0/2318744613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d20b7a700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c13eac0 0x7f3d1c0b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1b7fe700 1 -- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c13eac0 msgr2=0x7f3d1c0b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1b7fe700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c13eac0 0x7f3d1c0b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1bfff700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d1c0ce140 0x7f3d1c1f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1b7fe700 1 -- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d1c0ce140 msgr2=0x7f3d1c1f01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1b7fe700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d1c0ce140 0x7f3d1c1f01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1b7fe700 1 -- 172.21.15.77:0/2318744613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d14009a70 con 0x7f3d1c15af10 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d1b7fe700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 0x7f3d1c13e580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00c570 tx=0x7f3d0c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:20.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c004c80 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c004de0 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c007310 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c13f2e0 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.132+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c13f8a0 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.133+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c141890 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.133+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3d0c007470 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.134+0000 7f3d18ff9700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3d0806b350 0x7f3d0806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.134+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d0c089830 con 0x7f3d1c15af10 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.134+0000 7f3d1bfff700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3d0806b350 0x7f3d0806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:20.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.134+0000 7f3d1bfff700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3d0806b350 0x7f3d0806d810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009950 tx=0x7f3d10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:20.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.138+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d1c141890 con 0x7f3d1c15af10 2024-01-15T14:35:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.375+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3d1c067f60 con 0x7f3d1c15af10 2024-01-15T14:35:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.376+0000 7f3d18ff9700 1 -- 172.21.15.77:0/2318744613 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3d0c054730 con 0x7f3d1c15af10 2024-01-15T14:35:20.377 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3d0806b350 msgr2=0x7f3d0806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3d0806b350 0x7f3d0806d810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009950 tx=0x7f3d10008040 comp rx=0 tx=0).stop 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 msgr2=0x7f3d1c13e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 0x7f3d1c13e580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00c570 tx=0x7f3d0c00c930 comp rx=0 tx=0).stop 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 shutdown_connections 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3d0806b350 0x7f3d0806d810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3d1c13eac0 0x7f3d1c0b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3d1c0ce140 0x7f3d1c1f01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 --2- 172.21.15.77:0/2318744613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d1c15af10 0x7f3d1c13e580 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:20.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 >> 172.21.15.77:0/2318744613 conn(0x7f3d1c0cd570 msgr2=0x7f3d1c13dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:20.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 shutdown_connections 2024-01-15T14:35:20.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:20.378+0000 7f3d225dd700 1 -- 172.21.15.77:0/2318744613 wait complete. 2024-01-15T14:35:20.391 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T14:35:20.391 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T14:35:20.391 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-01-15T14:35:20.391 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T14:35:20.391 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T14:35:20.391 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T14:35:20.392 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T14:35:20.392 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T14:35:20.392 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T14:35:20.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.392 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-01-15T14:35:20.409 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-01-15T14:35:20.410 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42489sec preferred_lft 42489sec 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::944:e5c1:f7e3:6288/64 scope link noprefixroute 2024-01-15T14:35:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:20.412 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T14:35:20.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.412 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:20.412 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:20.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-01-15T14:35:20.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-01-15T14:35:20.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-01-15T14:35:20.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T14:35:20.413 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:20.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.576 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T14:35:20.605 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-15T14:35:20.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.607 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-01-15T14:35:20.687 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-01-15T14:35:20.687 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T14:35:20.687 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T14:35:20.687 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T14:35:20.687 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T14:35:20.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.688 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:20.688 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:20.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T14:35:20.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T14:35:20.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T14:35:20.688 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:20.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.931 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-15T14:35:20.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:20.949 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-01-15T14:35:21.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.007 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:21.007 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:21.007 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T14:35:21.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-15T14:35:21.008 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:21.104 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-15T14:35:21.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.104 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:21.104 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:21.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-15T14:35:21.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T14:35:21.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-15T14:35:21.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-15T14:35:21.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-15T14:35:21.105 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:21.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.302 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:21.302 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:21.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-01-15T14:35:21.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T14:35:21.302 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:21.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:21.400 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T14:35:21.417 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T14:35:21.418 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:21.418 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T14:35:21.475 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-01-15T14:35:21.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:21.561 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:21.561 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=a5eae108-559b-4366-a994-4d2898914163" will pass to kernel 2024-01-15T14:35:21.666 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T14:35:21.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T14:35:21.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.694 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:21.783 INFO:teuthology.orchestra.run.smithi077.stdout:client4591 2024-01-15T14:35:21.784 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:21.784 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:21.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.802 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:21.890 INFO:teuthology.orchestra.run.smithi077.stdout:client4591 2024-01-15T14:35:21.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.890 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:21.937 INFO:teuthology.orchestra.run.smithi077.stdout:client4591 2024-01-15T14:35:21.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:21.937 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a5eae108-559b-4366-a994-4d2898914163.client4591/status') 2024-01-15T14:35:21.990 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.4591 (3)172.21.15.77:0/3412064132 2024-01-15T14:35:21.990 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2024-01-15T14:35:21.990 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2024-01-15T14:35:21.990 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2024-01-15T14:35:21.991 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 3.6029e-05 s, 1.9 MB/s 2024-01-15T14:35:21.992 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-01-15T14:35:21.992 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T14:35:21.992 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T14:35:21.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T14:35:22.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.113+0000 7f3b8b0cd700 1 Processor -- start 2024-01-15T14:35:22.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.113+0000 7f3b8b0cd700 1 -- start start 2024-01-15T14:35:22.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.113+0000 7f3b8b0cd700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3b841446a0 0x7f3b840b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:22.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.113+0000 7f3b8b0cd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b840af950 con 0x7f3b8406c710 2024-01-15T14:35:22.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.113+0000 7f3b8b0cd700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b840afc30 con 0x7f3b840adf00 2024-01-15T14:35:22.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.113+0000 7f3b8b0cd700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b84144a80 con 0x7f3b841446a0 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b83fff700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3b840adf00 0x7f3b8406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:46704/0 (socket says 172.21.15.77:46704) 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b88e69700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3b841446a0 0x7f3b840b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b83fff700 1 -- 172.21.15.77:0/3405104173 learned_addr learned my addr 172.21.15.77:0/3405104173 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447741951 0 0) 0x7f3b840af950 con 0x7f3b8406c710 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b740035b0 con 0x7f3b8406c710 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684449148 0 0) 0x7f3b840afc30 con 0x7f3b840adf00 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b840af950 con 0x7f3b840adf00 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b88e69700 1 -- 172.21.15.77:0/3405104173 >> v1:172.21.15.159:6789/0 conn(0x7f3b840adf00 legacy=0x7f3b8406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b88e69700 1 -- 172.21.15.77:0/3405104173 >> v1:172.21.15.77:6789/0 conn(0x7f3b8406c710 legacy=0x7f3b840b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.114+0000 7f3b88e69700 1 -- 172.21.15.77:0/3405104173 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840aff10 con 0x7f3b841446a0 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b88e69700 1 --2- 172.21.15.77:0/3405104173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b841446a0 0x7f3b840b5ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009840 tx=0x7f3b78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5887b64389809b6b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78011830 con 0x7f3b841446a0 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b78011990 con 0x7f3b841446a0 2024-01-15T14:35:22.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b837fe700 1 -- 172.21.15.77:0/3405104173 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78011d00 con 0x7f3b841446a0 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/3405104173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b841446a0 msgr2=0x7f3b840b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/3405104173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b841446a0 0x7f3b840b5ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009840 tx=0x7f3b78009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/3405104173 shutdown_connections 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/3405104173 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b841446a0 0x7f3b840b5ae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/3405104173 >> 172.21.15.77:0/3405104173 conn(0x7f3b84071b90 msgr2=0x7f3b84071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/3405104173 shutdown_connections 2024-01-15T14:35:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.115+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/3405104173 wait complete. 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.117+0000 7f3b8b0cd700 1 Processor -- start 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.117+0000 7f3b8b0cd700 1 -- start start 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.117+0000 7f3b8b0cd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8406c710 0x7f3b84138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.117+0000 7f3b8b0cd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b840adf00 0x7f3b84136670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8b0cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 0x7f3b84155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8b0cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8413e4f0 con 0x7f3b841446a0 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8b0cd700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8413d450 con 0x7f3b840adf00 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8b0cd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8413d730 con 0x7f3b8406c710 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 0x7f3b84155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 0x7f3b84155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40658/0 (socket says 172.21.15.77:40658) 2024-01-15T14:35:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 -- 172.21.15.77:0/507034891 learned_addr learned my addr 172.21.15.77:0/507034891 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b83fff700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b840adf00 0x7f3b84136670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b88e69700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8406c710 0x7f3b84138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 -- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8406c710 msgr2=0x7f3b84138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8406c710 0x7f3b84138680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 -- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b840adf00 msgr2=0x7f3b84136670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b840adf00 0x7f3b84136670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 -- 172.21.15.77:0/507034891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b780094a0 con 0x7f3b841446a0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b83fff700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b840adf00 0x7f3b84136670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8966a700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 0x7f3b84155680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00cba0 tx=0x7f3b7c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c003c00 con 0x7f3b841446a0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7c003d60 con 0x7f3b841446a0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8413da10 con 0x7f3b841446a0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c00aba0 con 0x7f3b841446a0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.118+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8413dfd0 con 0x7f3b841446a0 2024-01-15T14:35:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.119+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b840753f0 con 0x7f3b841446a0 2024-01-15T14:35:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.120+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3b7c00ad00 con 0x7f3b841446a0 2024-01-15T14:35:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.120+0000 7f3b817fa700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b6806b350 0x7f3b6806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.120+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b7c089c90 con 0x7f3b841446a0 2024-01-15T14:35:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.120+0000 7f3b88e69700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b6806b350 0x7f3b6806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.120+0000 7f3b88e69700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b6806b350 0x7f3b6806d810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009fd0 tx=0x7f3b7801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:22.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.125+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b7c054b10 con 0x7f3b841446a0 2024-01-15T14:35:22.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.361+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3b8406e380 con 0x7f3b841446a0 2024-01-15T14:35:22.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.361+0000 7f3b817fa700 1 -- 172.21.15.77:0/507034891 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3b7c058130 con 0x7f3b841446a0 2024-01-15T14:35:22.362 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b6806b350 msgr2=0x7f3b6806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b6806b350 0x7f3b6806d810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009fd0 tx=0x7f3b7801c040 comp rx=0 tx=0).stop 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 msgr2=0x7f3b84155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 0x7f3b84155680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00cba0 tx=0x7f3b7c00cf60 comp rx=0 tx=0).stop 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 shutdown_connections 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3b6806b350 0x7f3b6806d810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3b8406c710 0x7f3b84138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b841446a0 0x7f3b84155680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 --2- 172.21.15.77:0/507034891 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3b840adf00 0x7f3b84136670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.363+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 >> 172.21.15.77:0/507034891 conn(0x7f3b84071b90 msgr2=0x7f3b840b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.364+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 shutdown_connections 2024-01-15T14:35:22.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:22.364+0000 7f3b8b0cd700 1 -- 172.21.15.77:0/507034891 wait complete. 2024-01-15T14:35:22.376 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T14:35:22.377 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-15T14:35:22.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.377 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-01-15T14:35:22.395 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42487sec preferred_lft 42487sec 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::944:e5c1:f7e3:6288/64 scope link noprefixroute 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether a2:d0:23:55:e3:05 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:22.396 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8863:bcff:fea3:4402/64 scope link 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether a2:d0:23:55:e3:05 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a0d0:23ff:fe55:e305/64 scope link tentative 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.397 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T14:35:22.461 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-15T14:35:22.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.463 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T14:35:22.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.523 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:22.523 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:22.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T14:35:22.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T14:35:22.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T14:35:22.523 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:22.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.630 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-15T14:35:22.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T14:35:22.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.648 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-01-15T14:35:22.705 INFO:teuthology.orchestra.run.smithi077.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-15T14:35:22.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.705 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:22.706 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:22.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T14:35:22.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-15T14:35:22.706 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:22.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T14:35:22.836 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:22.836 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:22.836 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:22.836 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether a6:b3:69:89:fc:44 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4b3:69ff:fe89:fc44/64 scope link 2024-01-15T14:35:22.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.926 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:35:22.927 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether a6:b3:69:89:fc:44 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T14:35:22.927 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T14:35:22.927 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.927 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4b3:69ff:fe89:fc44/64 scope link 2024-01-15T14:35:22.927 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:22.943 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-15T14:35:22.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-15T14:35:22.944 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:23.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.196 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:35:23.196 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:35:23.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.1 up 2024-01-15T14:35:23.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T14:35:23.196 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:35:23.294 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:23.294 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-15T14:35:23.310 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-15T14:35:23.310 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:23.310 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-15T14:35:23.368 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-01-15T14:35:23.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:23.454 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:23.454 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 2@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:23.474 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:23.492 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:23.492 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 2@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-01-15T14:35:23.493 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,key=2,mds_namespace=cephfs,fsid=a5eae108-559b-4366-a994-4d2898914163" will pass to kernel 2024-01-15T14:35:23.495 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T14:35:23.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-15T14:35:23.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.521 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:23.594 INFO:teuthology.orchestra.run.smithi077.stdout:client4603 2024-01-15T14:35:23.595 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:23.595 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:23.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.652 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:23.725 INFO:teuthology.orchestra.run.smithi077.stdout:client4603 2024-01-15T14:35:23.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.726 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:23.798 INFO:teuthology.orchestra.run.smithi077.stdout:client4603 2024-01-15T14:35:23.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:23.798 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a5eae108-559b-4366-a994-4d2898914163.client4603/status') 2024-01-15T14:35:23.878 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.4603 (3)172.21.15.77:0/2641129213 2024-01-15T14:35:23.878 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2024-01-15T14:35:23.878 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2024-01-15T14:35:23.878 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2024-01-15T14:35:23.878 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 3.8231e-05 s, 1.8 MB/s 2024-01-15T14:35:23.879 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-01-15T14:35:23.880 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T14:35:23.880 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T14:35:23.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T14:35:24.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.040+0000 7f09d281a700 1 Processor -- start 2024-01-15T14:35:24.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.040+0000 7f09d281a700 1 -- start start 2024-01-15T14:35:24.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09d281a700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f09cc15e1e0 0x7f09cc0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:24.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09d281a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d0f80 0x7f09cc0b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:24.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09d281a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0d20b0 con 0x7f09cc0d3a00 2024-01-15T14:35:24.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09d281a700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc0d3150 con 0x7f09cc15e1e0 2024-01-15T14:35:24.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09d281a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc15e5c0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09cb7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f09cc0d3a00 0x7f09cc0d0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34936/0 (socket says 172.21.15.77:34936) 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09cb7fe700 1 -- 172.21.15.77:0/3381343598 learned_addr learned my addr 172.21.15.77:0/3381343598 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09cbfff700 1 --2- 172.21.15.77:0/3381343598 >> v2:172.21.15.159:3300/0 conn(0x7f09cc15e1e0 0x7f09cc0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.041+0000 7f09d0db7700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d0f80 0x7f09cc0b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09caffd700 1 -- 172.21.15.77:0/3381343598 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577490080 0 0) 0x7f09cc0d20b0 con 0x7f09cc0d3a00 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09caffd700 1 -- 172.21.15.77:0/3381343598 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b40035b0 con 0x7f09cc0d3a00 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09cbfff700 1 -- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d0f80 msgr2=0x7f09cc0b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09cbfff700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d0f80 0x7f09cc0b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09cbfff700 1 -- 172.21.15.77:0/3381343598 >> v1:172.21.15.77:6789/0 conn(0x7f09cc0d3a00 legacy=0x7f09cc0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09cbfff700 1 -- 172.21.15.77:0/3381343598 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0d2390 con 0x7f09cc15e1e0 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09d0db7700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d0f80 0x7f09cc0b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09cbfff700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0d34c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009870 tx=0x7f09bc009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d01f48c22b6e1f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09caffd700 1 -- 172.21.15.77:0/3381343598 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011830 con 0x7f09cc15e1e0 2024-01-15T14:35:24.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09caffd700 1 -- 172.21.15.77:0/3381343598 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc011990 con 0x7f09cc15e1e0 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.042+0000 7f09caffd700 1 -- 172.21.15.77:0/3381343598 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011d00 con 0x7f09cc15e1e0 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 -- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 msgr2=0x7f09cc0d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0d34c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009870 tx=0x7f09bc009b80 comp rx=0 tx=0).stop 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 -- 172.21.15.77:0/3381343598 shutdown_connections 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d0f80 0x7f09cc0b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 --2- 172.21.15.77:0/3381343598 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0d34c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 -- 172.21.15.77:0/3381343598 >> 172.21.15.77:0/3381343598 conn(0x7f09cc0b5b50 msgr2=0x7f09cc0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 -- 172.21.15.77:0/3381343598 shutdown_connections 2024-01-15T14:35:24.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.043+0000 7f09d281a700 1 -- 172.21.15.77:0/3381343598 wait complete. 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.044+0000 7f09d281a700 1 Processor -- start 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.044+0000 7f09d281a700 1 -- start start 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 0x7f09cc1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d3a00 0x7f09cc1515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc156fa0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc157840 con 0x7f09cc15e1e0 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc161420 con 0x7f09cc0d3a00 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 0x7f09cc1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 0x7f09cc1582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40668/0 (socket says 172.21.15.77:40668) 2024-01-15T14:35:24.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 -- 172.21.15.77:0/1041149641 learned_addr learned my addr 172.21.15.77:0/1041149641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d0db7700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cb7fe700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d3a00 0x7f09cc1515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 -- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d3a00 msgr2=0x7f09cc1515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d3a00 0x7f09cc1515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 -- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 msgr2=0x7f09cc0ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 -- 172.21.15.77:0/1041149641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c4009a70 con 0x7f09cc0d0f80 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d0db7700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cb7fe700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d3a00 0x7f09cc1515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09cbfff700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 0x7f09cc1582e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0097c0 tx=0x7f09bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0045c0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc01e070 con 0x7f09cc0d0f80 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.045+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc0094a0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.046+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0038e0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.046+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc1574d0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.046+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc0751e0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.047+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09bc003a40 con 0x7f09cc0d0f80 2024-01-15T14:35:24.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.047+0000 7f09c8ff9700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f09b806b350 0x7f09b806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:24.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.047+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09bc08aa30 con 0x7f09cc0d0f80 2024-01-15T14:35:24.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.047+0000 7f09cb7fe700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f09b806b350 0x7f09b806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:24.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.048+0000 7f09cb7fe700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f09b806b350 0x7f09b806d810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0ccc50 tx=0x7f09c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:24.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.052+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09bc055930 con 0x7f09cc0d0f80 2024-01-15T14:35:24.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.290+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f09cc0b37c0 con 0x7f09cc0d0f80 2024-01-15T14:35:24.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.290+0000 7f09c8ff9700 1 -- 172.21.15.77:0/1041149641 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f09bc015020 con 0x7f09cc0d0f80 2024-01-15T14:35:24.291 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f09b806b350 msgr2=0x7f09b806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f09b806b350 0x7f09b806d810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0ccc50 tx=0x7f09c0009040 comp rx=0 tx=0).stop 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 msgr2=0x7f09cc1582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 0x7f09cc1582e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0097c0 tx=0x7f09bc004750 comp rx=0 tx=0).stop 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 shutdown_connections 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f09b806b350 0x7f09b806d810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f09cc15e1e0 0x7f09cc0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09cc0d0f80 0x7f09cc1582e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 --2- 172.21.15.77:0/1041149641 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f09cc0d3a00 0x7f09cc1515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 >> 172.21.15.77:0/1041149641 conn(0x7f09cc0b5b50 msgr2=0x7f09cc0b84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 shutdown_connections 2024-01-15T14:35:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:24.293+0000 7f09d281a700 1 -- 172.21.15.77:0/1041149641 wait complete. 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T14:35:24.306 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T14:35:24.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.307 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:24.328 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42488sec preferred_lft 42488sec 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::6b97:3ef3:99d1:2a4/64 scope link noprefixroute 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:24.329 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T14:35:24.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.329 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:24.329 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:24.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add name ceph-brx type bridge 2024-01-15T14:35:24.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip addr flush dev ceph-brx 2024-01-15T14:35:24.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set ceph-brx up 2024-01-15T14:35:24.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T14:35:24.330 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:24.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.509 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T14:35:24.537 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-01-15T14:35:24.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.539 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-01-15T14:35:24.608 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-01-15T14:35:24.608 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T14:35:24.608 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T14:35:24.608 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T14:35:24.608 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T14:35:24.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.609 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:24.609 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:24.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T14:35:24.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T14:35:24.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T14:35:24.609 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:24.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.859 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-01-15T14:35:24.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.876 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list-id 2024-01-15T14:35:24.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:24.933 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:24.933 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:24.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T14:35:24.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-15T14:35:24.933 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:25.028 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-15T14:35:25.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.028 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:25.028 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:25.028 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-15T14:35:25.028 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T14:35:25.029 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-15T14:35:25.029 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-15T14:35:25.029 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-15T14:35:25.029 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:25.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.291 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:25.291 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:25.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-01-15T14:35:25.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T14:35:25.291 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:25.388 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:25.388 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T14:35:25.405 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T14:35:25.405 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:25.405 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T14:35:25.461 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-01-15T14:35:25.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:25.547 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:25.548 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:25.655 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:25.655 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:25.655 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:25.655 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 1@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:25.656 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:25.656 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:25.656 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:25.656 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:25.673 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 1@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-01-15T14:35:25.674 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=a5eae108-559b-4366-a994-4d2898914163" will pass to kernel 2024-01-15T14:35:25.677 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T14:35:25.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.677 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T14:35:25.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.705 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:25.777 INFO:teuthology.orchestra.run.smithi159.stdout:client4475 2024-01-15T14:35:25.778 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:25.778 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:25.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.836 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:25.908 INFO:teuthology.orchestra.run.smithi159.stdout:client4475 2024-01-15T14:35:25.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.909 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:25.980 INFO:teuthology.orchestra.run.smithi159.stdout:client4475 2024-01-15T14:35:25.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:25.981 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a5eae108-559b-4366-a994-4d2898914163.client4475/status') 2024-01-15T14:35:26.060 INFO:teuthology.orchestra.run.smithi159.stdout:instance: client.4475 (3)192.168.0.1:0/2488305615 2024-01-15T14:35:26.060 INFO:teuthology.orchestra.run.smithi159.stdout:blocklisted: false 2024-01-15T14:35:26.061 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records in 2024-01-15T14:35:26.061 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records out 2024-01-15T14:35:26.061 INFO:teuthology.orchestra.run.smithi159.stderr:69 bytes copied, 2.1581e-05 s, 3.2 MB/s 2024-01-15T14:35:26.062 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-01-15T14:35:26.062 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T14:35:26.062 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T14:35:26.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.196+0000 7fb6bdda6700 1 Processor -- start 2024-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.196+0000 7fb6bdda6700 1 -- start start 2024-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.196+0000 7fb6bdda6700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb6b815e2a0 0x7fb6b806ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.196+0000 7fb6bdda6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b806e7f0 con 0x7fb6b80ce090 2024-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6bdda6700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b806dec0 con 0x7fb6b8073f90 2024-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6bdda6700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b806e1a0 con 0x7fb6b815e2a0 2024-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b6ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb6b80ce090 0x7fb6b806ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34942/0 (socket says 172.21.15.77:34942) 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b6ffd700 1 -- 172.21.15.77:0/79777940 learned_addr learned my addr 172.21.15.77:0/79777940 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b77fe700 1 --2- 172.21.15.77:0/79777940 >> v2:172.21.15.159:3300/0 conn(0x7fb6b815e2a0 0x7fb6b806ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b67fc700 1 -- 172.21.15.77:0/79777940 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190374847 0 0) 0x7fb6b806e7f0 con 0x7fb6b80ce090 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b67fc700 1 -- 172.21.15.77:0/79777940 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a40035b0 con 0x7fb6b80ce090 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b77fe700 1 -- 172.21.15.77:0/79777940 >> v1:172.21.15.159:6789/0 conn(0x7fb6b8073f90 legacy=0x7fb6b80cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b77fe700 1 -- 172.21.15.77:0/79777940 >> v1:172.21.15.77:6789/0 conn(0x7fb6b80ce090 legacy=0x7fb6b806ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b77fe700 1 -- 172.21.15.77:0/79777940 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b806e480 con 0x7fb6b815e2a0 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b77fe700 1 --2- 172.21.15.77:0/79777940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b815e2a0 0x7fb6b806ba70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009840 tx=0x7fb6a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58bb11bbe027b5f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.197+0000 7fb6b67fc700 1 -- 172.21.15.77:0/79777940 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8011830 con 0x7fb6b815e2a0 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6b67fc700 1 -- 172.21.15.77:0/79777940 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a8011990 con 0x7fb6b815e2a0 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6b67fc700 1 -- 172.21.15.77:0/79777940 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8011d20 con 0x7fb6b815e2a0 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 -- 172.21.15.77:0/79777940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b815e2a0 msgr2=0x7fb6b806ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/79777940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b815e2a0 0x7fb6b806ba70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009840 tx=0x7fb6a8009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 -- 172.21.15.77:0/79777940 shutdown_connections 2024-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/79777940 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b815e2a0 0x7fb6b806ba70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 -- 172.21.15.77:0/79777940 >> 172.21.15.77:0/79777940 conn(0x7fb6b80cd570 msgr2=0x7fb6b80cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 -- 172.21.15.77:0/79777940 shutdown_connections 2024-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.198+0000 7fb6bdda6700 1 -- 172.21.15.77:0/79777940 wait complete. 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.199+0000 7fb6bdda6700 1 Processor -- start 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 -- start start 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b8073f90 0x7fb6b8156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80ce090 0x7fb6b8158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 0x7fb6b81f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b8156840 con 0x7fb6b815e2a0 2024-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b8156570 con 0x7fb6b80ce090 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6bdda6700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b8157f40 con 0x7fb6b8073f90 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 0x7fb6b81f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 0x7fb6b81f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40676/0 (socket says 172.21.15.77:40676) 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6b7fff700 1 -- 172.21.15.77:0/2700048653 learned_addr learned my addr 172.21.15.77:0/2700048653 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6b77fe700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b8073f90 0x7fb6b8156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.200+0000 7fb6b6ffd700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80ce090 0x7fb6b8158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6b7fff700 1 -- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b8073f90 msgr2=0x7fb6b8156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6b7fff700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b8073f90 0x7fb6b8156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6b7fff700 1 -- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80ce090 msgr2=0x7fb6b8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6b7fff700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80ce090 0x7fb6b8158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6b7fff700 1 -- 172.21.15.77:0/2700048653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a80094a0 con 0x7fb6b815e2a0 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6b7fff700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 0x7fb6b81f70b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000cc50 tx=0x7fb6b000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0003c00 con 0x7fb6b815e2a0 2024-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b0003d60 con 0x7fb6b815e2a0 2024-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000aab0 con 0x7fb6b815e2a0 2024-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b8158220 con 0x7fb6b815e2a0 2024-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.201+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b81587e0 con 0x7fb6b815e2a0 2024-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.202+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b8067f60 con 0x7fb6b815e2a0 2024-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.202+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb6b000ac10 con 0x7fb6b815e2a0 2024-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.203+0000 7fb6affff700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb69806b350 0x7fb69806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.203+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6b0089b70 con 0x7fb6b815e2a0 2024-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.203+0000 7fb6b77fe700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb69806b350 0x7fb69806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.203+0000 7fb6b77fe700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb69806b350 0x7fb69806d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80090b0 tx=0x7fb6a801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.207+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6b00549e0 con 0x7fb6b815e2a0 2024-01-15T14:35:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.441+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb6b8072e00 con 0x7fb6b815e2a0 2024-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.442+0000 7fb6affff700 1 -- 172.21.15.77:0/2700048653 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb6b0058000 con 0x7fb6b815e2a0 2024-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb69806b350 msgr2=0x7fb69806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb69806b350 0x7fb69806d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80090b0 tx=0x7fb6a801c040 comp rx=0 tx=0).stop 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 msgr2=0x7fb6b81f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 0x7fb6b81f70b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000cc50 tx=0x7fb6b000cf60 comp rx=0 tx=0).stop 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 shutdown_connections 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb69806b350 0x7fb69806d810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb6b8073f90 0x7fb6b8156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6b815e2a0 0x7fb6b81f70b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 --2- 172.21.15.77:0/2700048653 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb6b80ce090 0x7fb6b8158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 >> 172.21.15.77:0/2700048653 conn(0x7fb6b80cd570 msgr2=0x7fb6b80b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 shutdown_connections 2024-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:26.444+0000 7fb6bdda6700 1 -- 172.21.15.77:0/2700048653 wait complete. 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T14:35:26.457 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-15T14:35:26.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.457 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:26.476 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42486sec preferred_lft 42486sec 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::6b97:3ef3:99d1:2a4/64 scope link noprefixroute 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether a6:ae:a0:c4:3e:57 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::b8cb:fff:fe98:f453/64 scope link 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T14:35:26.477 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether a6:ae:a0:c4:3e:57 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T14:35:26.478 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::a4ae:a0ff:fec4:3e57/64 scope link tentative 2024-01-15T14:35:26.478 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.478 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T14:35:26.542 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-01-15T14:35:26.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.543 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-01-15T14:35:26.602 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-01-15T14:35:26.602 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T14:35:26.602 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T14:35:26.602 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T14:35:26.602 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T14:35:26.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.603 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:26.603 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:26.603 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T14:35:26.603 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T14:35:26.603 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T14:35:26.603 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:26.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.708 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-01-15T14:35:26.725 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T14:35:26.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.725 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list-id 2024-01-15T14:35:26.782 INFO:teuthology.orchestra.run.smithi159.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-15T14:35:26.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.783 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:26.783 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:26.783 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T14:35:26.783 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-15T14:35:26.783 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:26.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.877 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T14:35:26.914 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:26.914 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:26.914 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 36:9f:07:5c:cc:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::349f:7ff:fe5c:cca3/64 scope link 2024-01-15T14:35:26.915 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:26.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:26.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T14:35:27.001 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 36:9f:07:5c:cc:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:27.002 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::349f:7ff:fe5c:cca3/64 scope link 2024-01-15T14:35:27.003 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:35:27.013 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-15T14:35:27.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-15T14:35:27.014 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:27.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.273 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:35:27.273 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:35:27.273 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.1 up 2024-01-15T14:35:27.273 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T14:35:27.273 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:35:27.371 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:27.371 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-15T14:35:27.388 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-15T14:35:27.389 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:35:27.389 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-15T14:35:27.447 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-01-15T14:35:27.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.448 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:27.534 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:27.534 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:27.553 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:27.553 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:27.554 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:27.554 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 3@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:27.554 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:27.554 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:27.554 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:27.554 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:27.571 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 3@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-01-15T14:35:27.572 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,key=3,mds_namespace=cephfs,fsid=a5eae108-559b-4366-a994-4d2898914163" will pass to kernel 2024-01-15T14:35:27.574 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T14:35:27.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.574 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-15T14:35:27.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.602 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:27.675 INFO:teuthology.orchestra.run.smithi159.stdout:client4621 2024-01-15T14:35:27.675 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:27.675 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:27.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.734 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:27.807 INFO:teuthology.orchestra.run.smithi159.stdout:client4621 2024-01-15T14:35:27.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.807 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:35:27.880 INFO:teuthology.orchestra.run.smithi159.stdout:client4621 2024-01-15T14:35:27.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:27.880 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a5eae108-559b-4366-a994-4d2898914163.client4621/status') 2024-01-15T14:35:27.961 INFO:teuthology.orchestra.run.smithi159.stdout:instance: client.4621 (3)192.168.0.2:0/832933294 2024-01-15T14:35:27.961 INFO:teuthology.orchestra.run.smithi159.stdout:blocklisted: false 2024-01-15T14:35:27.961 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records in 2024-01-15T14:35:27.961 INFO:teuthology.orchestra.run.smithi159.stderr:0+1 records out 2024-01-15T14:35:27.962 INFO:teuthology.orchestra.run.smithi159.stderr:68 bytes copied, 2.1929e-05 s, 3.1 MB/s 2024-01-15T14:35:27.963 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-15T14:35:27.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:28.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.114+0000 7fd9a4a42700 1 Processor -- start 2024-01-15T14:35:28.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.114+0000 7fd9a4a42700 1 -- start start 2024-01-15T14:35:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd9a4a42700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd9a015e060 0x7fd9a00ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd9a4a42700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9a015e440 0x7fd9a01622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd9a4a42700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a0161b70 con 0x7fd9a015e8c0 2024-01-15T14:35:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd9a4a42700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a015ed60 con 0x7fd9a015e440 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd9a4a42700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a015f040 con 0x7fd9a015e060 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99dd9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9a015e440 0x7fd9a01622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99ed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd9a015e8c0 0x7fd9a0073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34958/0 (socket says 172.21.15.77:34958) 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99ed9d700 1 -- 172.21.15.77:0/764335638 learned_addr learned my addr 172.21.15.77:0/764335638 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99e59c700 1 --2- 172.21.15.77:0/764335638 >> v2:172.21.15.159:3300/0 conn(0x7fd9a015e060 0x7fd9a00ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99dd9b700 1 -- 172.21.15.77:0/764335638 >> v2:172.21.15.159:3300/0 conn(0x7fd9a015e060 msgr2=0x7fd9a00ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99dd9b700 1 --2- 172.21.15.77:0/764335638 >> v2:172.21.15.159:3300/0 conn(0x7fd9a015e060 0x7fd9a00ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99dd9b700 1 -- 172.21.15.77:0/764335638 >> v1:172.21.15.77:6789/0 conn(0x7fd9a015e8c0 legacy=0x7fd9a0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99dd9b700 1 -- 172.21.15.77:0/764335638 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a015f320 con 0x7fd9a015e440 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.115+0000 7fd99d59a700 1 -- 172.21.15.77:0/764335638 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247133747 0 0) 0x7fd9a0161b70 con 0x7fd9a015e8c0 2024-01-15T14:35:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd99dd9b700 1 --2- 172.21.15.77:0/764335638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e440 0x7fd9a01622e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0072cf0 tx=0x7fd990009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caec550e97da6254 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd99d59a700 1 -- 172.21.15.77:0/764335638 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd990011830 con 0x7fd9a015e440 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd99d59a700 1 -- 172.21.15.77:0/764335638 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd990011990 con 0x7fd9a015e440 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd99d59a700 1 -- 172.21.15.77:0/764335638 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd990011d20 con 0x7fd9a015e440 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 -- 172.21.15.77:0/764335638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e440 msgr2=0x7fd9a01622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/764335638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e440 0x7fd9a01622e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0072cf0 tx=0x7fd990009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 -- 172.21.15.77:0/764335638 shutdown_connections 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/764335638 >> v2:172.21.15.159:3300/0 conn(0x7fd9a015e060 0x7fd9a00ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/764335638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e440 0x7fd9a01622e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 -- 172.21.15.77:0/764335638 >> 172.21.15.77:0/764335638 conn(0x7fd9a00b3190 msgr2=0x7fd9a00d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 -- 172.21.15.77:0/764335638 shutdown_connections 2024-01-15T14:35:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.116+0000 7fd9a4a42700 1 -- 172.21.15.77:0/764335638 wait complete. 2024-01-15T14:35:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.118+0000 7fd9a4a42700 1 Processor -- start 2024-01-15T14:35:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.118+0000 7fd9a4a42700 1 -- start start 2024-01-15T14:35:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 0x7fd9a01479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9a015e440 0x7fd9a01f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9a015e8c0 0x7fd9a0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a01f7f10 con 0x7fd9a015e060 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a01561f0 con 0x7fd9a015e440 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a0143ff0 con 0x7fd9a015e8c0 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 0x7fd9a01479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 0x7fd9a01479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40690/0 (socket says 172.21.15.77:40690) 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 -- 172.21.15.77:0/511336628 learned_addr learned my addr 172.21.15.77:0/511336628 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99ed9d700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9a015e8c0 0x7fd9a0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99dd9b700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9a015e440 0x7fd9a01f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 -- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9a015e8c0 msgr2=0x7fd9a0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9a015e8c0 0x7fd9a0155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 -- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9a015e440 msgr2=0x7fd9a01f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9a015e440 0x7fd9a01f79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 -- 172.21.15.77:0/511336628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9900094a0 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd99e59c700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 0x7fd9a01479a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e810 tx=0x7fd99400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99400c070 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994013610 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a01442d0 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940139c0 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.119+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a0144890 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.120+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a0067f60 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.121+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd994013b20 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.121+0000 7fd98effd700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd98006b350 0x7fd98006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.121+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd994089900 con 0x7fd9a015e060 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.121+0000 7fd99dd9b700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd98006b350 0x7fd98006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.122+0000 7fd99dd9b700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd98006b350 0x7fd98006d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd9900090b0 tx=0x7fd990008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.126+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd994054780 con 0x7fd9a015e060 2024-01-15T14:35:28.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.321+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9a00d1880 con 0x7fd9a015e060 2024-01-15T14:35:28.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.322+0000 7fd98effd700 1 -- 172.21.15.77:0/511336628 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd994057da0 con 0x7fd9a015e060 2024-01-15T14:35:28.324 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:28.324 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd98006b350 msgr2=0x7fd98006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd98006b350 0x7fd98006d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd9900090b0 tx=0x7fd990008030 comp rx=0 tx=0).stop 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 msgr2=0x7fd9a01479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 0x7fd9a01479a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e810 tx=0x7fd99400eb20 comp rx=0 tx=0).stop 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 shutdown_connections 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd98006b350 0x7fd98006d810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd9a015e8c0 0x7fd9a0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a015e060 0x7fd9a01479a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 --2- 172.21.15.77:0/511336628 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd9a015e440 0x7fd9a01f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 >> 172.21.15.77:0/511336628 conn(0x7fd9a00b3190 msgr2=0x7fd9a00d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 shutdown_connections 2024-01-15T14:35:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.324+0000 7fd9a4a42700 1 -- 172.21.15.77:0/511336628 wait complete. 2024-01-15T14:35:28.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:28.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:28.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.461+0000 7fe4e6e60700 1 Processor -- start 2024-01-15T14:35:28.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.461+0000 7fe4e6e60700 1 -- start start 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4e6e60700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe4e015e150 0x7fe4e0161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4e6e60700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4e00b9860 0x7fe4e00ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4e6e60700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4e6e60700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e00cf500 con 0x7fe4e00b9860 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4e6e60700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e00d22f0 con 0x7fe4e015e150 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4e6e60700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0162110 con 0x7fe4e00cf070 2024-01-15T14:35:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4dffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4e00b9860 0x7fe4e00ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4dffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4e00b9860 0x7fe4e00ceb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40692/0 (socket says 172.21.15.77:40692) 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.462+0000 7fe4dffff700 1 -- 172.21.15.77:0/4106160360 learned_addr learned my addr 172.21.15.77:0/4106160360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4e4bfc700 1 --2- 172.21.15.77:0/4106160360 >> v2:172.21.15.159:3300/0 conn(0x7fe4e015e150 0x7fe4e0161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4e53fd700 1 --2- 172.21.15.77:0/4106160360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4dffff700 1 -- 172.21.15.77:0/4106160360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 msgr2=0x7fe4e00b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4dffff700 1 --2- 172.21.15.77:0/4106160360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4dffff700 1 -- 172.21.15.77:0/4106160360 >> v2:172.21.15.159:3300/0 conn(0x7fe4e015e150 msgr2=0x7fe4e0161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4dffff700 1 --2- 172.21.15.77:0/4106160360 >> v2:172.21.15.159:3300/0 conn(0x7fe4e015e150 0x7fe4e0161b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4dffff700 1 -- 172.21.15.77:0/4106160360 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e01623f0 con 0x7fe4e00b9860 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4e4bfc700 1 --2- 172.21.15.77:0/4106160360 >> v2:172.21.15.159:3300/0 conn(0x7fe4e015e150 0x7fe4e0161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4dffff700 1 --2- 172.21.15.77:0/4106160360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e00ceb30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00d0840 tx=0x7fe4cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8583dcc515ba09 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4df7fe700 1 -- 172.21.15.77:0/4106160360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00b870 con 0x7fe4e00b9860 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4df7fe700 1 -- 172.21.15.77:0/4106160360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc00b9d0 con 0x7fe4e00b9860 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.463+0000 7fe4df7fe700 1 -- 172.21.15.77:0/4106160360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00bd60 con 0x7fe4e00b9860 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 -- 172.21.15.77:0/4106160360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 msgr2=0x7fe4e00ceb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/4106160360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e00ceb30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00d0840 tx=0x7fe4cc009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 -- 172.21.15.77:0/4106160360 shutdown_connections 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/4106160360 >> v2:172.21.15.159:3300/0 conn(0x7fe4e015e150 0x7fe4e0161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/4106160360 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/4106160360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e00ceb30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 -- 172.21.15.77:0/4106160360 >> 172.21.15.77:0/4106160360 conn(0x7fe4e006b620 msgr2=0x7fe4e0071610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 -- 172.21.15.77:0/4106160360 shutdown_connections 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.464+0000 7fe4e6e60700 1 -- 172.21.15.77:0/4106160360 wait complete. 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.465+0000 7fe4e6e60700 1 Processor -- start 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.465+0000 7fe4e6e60700 1 -- start start 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e6e60700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e01f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e6e60700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e6e60700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e015e150 0x7fe4e00b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e6e60700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00aa930 con 0x7fe4e00b9860 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e6e60700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00aac10 con 0x7fe4e015e150 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e6e60700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00aaef0 con 0x7fe4e00cf070 2024-01-15T14:35:28.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e01f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e01f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40694/0 (socket says 172.21.15.77:40694) 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 -- 172.21.15.77:0/3774054449 learned_addr learned my addr 172.21.15.77:0/3774054449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e53fd700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e015e150 0x7fe4e00b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4dffff700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 -- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 msgr2=0x7fe4e00b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 -- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e015e150 msgr2=0x7fe4e00b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e015e150 0x7fe4e00b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 -- 172.21.15.77:0/3774054449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc009550 con 0x7fe4e00b9860 2024-01-15T14:35:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4e4bfc700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e01f4a70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400c7f0 tx=0x7fe4d400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4011800 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.466+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4011960 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.467+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4011cf0 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.467+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e00ab1d0 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.467+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e01f4fb0 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.467+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e0069020 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.468+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe4d4018360 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.468+0000 7fe4dd7fa700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe4c406b350 0x7fe4c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.468+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4d40899f0 con 0x7fe4e00b9860 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.468+0000 7fe4dffff700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe4c406b350 0x7fe4c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.469+0000 7fe4dffff700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe4c406b350 0x7fe4c406d810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000c00 tx=0x7fe4cc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.473+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d4054860 con 0x7fe4e00b9860 2024-01-15T14:35:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.663+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe4e0161820 con 0x7fe4e00b9860 2024-01-15T14:35:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.664+0000 7fe4dd7fa700 1 -- 172.21.15.77:0/3774054449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe4d401f020 con 0x7fe4e00b9860 2024-01-15T14:35:28.665 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:28.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe4c406b350 msgr2=0x7fe4c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe4c406b350 0x7fe4c406d810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000c00 tx=0x7fe4cc008030 comp rx=0 tx=0).stop 2024-01-15T14:35:28.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 msgr2=0x7fe4e01f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e01f4a70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400c7f0 tx=0x7fe4d400cb00 comp rx=0 tx=0).stop 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 shutdown_connections 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe4c406b350 0x7fe4c406d810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe4e015e150 0x7fe4e00b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e00b9860 0x7fe4e01f4a70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 --2- 172.21.15.77:0/3774054449 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe4e00cf070 0x7fe4e00b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 >> 172.21.15.77:0/3774054449 conn(0x7fe4e006b620 msgr2=0x7fe4e015cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 shutdown_connections 2024-01-15T14:35:28.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.666+0000 7fe4e6e60700 1 -- 172.21.15.77:0/3774054449 wait complete. 2024-01-15T14:35:28.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:28.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.803+0000 7f6529c4f700 1 Processor -- start 2024-01-15T14:35:28.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.803+0000 7f6529c4f700 1 -- start start 2024-01-15T14:35:28.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6529c4f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65240d4590 0x7f65240d3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6529c4f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65240d23e0 con 0x7f65240cf110 2024-01-15T14:35:28.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6529c4f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65240d5010 con 0x7f652415e110 2024-01-15T14:35:28.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6529c4f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65240cf570 con 0x7f65240d4590 2024-01-15T14:35:28.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65240d4590 0x7f65240d3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65240d4590 0x7f65240d3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40706/0 (socket says 172.21.15.77:40706) 2024-01-15T14:35:28.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 -- 172.21.15.77:0/1840543629 learned_addr learned my addr 172.21.15.77:0/1840543629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:28.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6522ffd700 1 --1- 172.21.15.77:0/1840543629 >> v1:172.21.15.77:6789/0 conn(0x7f65240cf110 0x7f65240d2870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34968/0 (socket says 172.21.15.77:34968) 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 -- 172.21.15.77:0/1840543629 >> v1:172.21.15.159:6789/0 conn(0x7f652415e110 legacy=0x7f65240b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 -- 172.21.15.77:0/1840543629 >> v1:172.21.15.77:6789/0 conn(0x7f65240cf110 legacy=0x7f65240d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f65227fc700 1 -- 172.21.15.77:0/1840543629 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169885574 0 0) 0x7f65240d23e0 con 0x7f65240cf110 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 -- 172.21.15.77:0/1840543629 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652415e4f0 con 0x7f65240d4590 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f6523fff700 1 --2- 172.21.15.77:0/1840543629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240d4590 0x7f65240d3530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f651c009e50 tx=0x7f651c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2e5ad727e3eef9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f65227fc700 1 -- 172.21.15.77:0/1840543629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00e070 con 0x7f65240d4590 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f65227fc700 1 -- 172.21.15.77:0/1840543629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c003a20 con 0x7f65240d4590 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.804+0000 7f65227fc700 1 -- 172.21.15.77:0/1840543629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c003dd0 con 0x7f65240d4590 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 -- 172.21.15.77:0/1840543629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240d4590 msgr2=0x7f65240d3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1840543629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240d4590 0x7f65240d3530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f651c009e50 tx=0x7f651c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 -- 172.21.15.77:0/1840543629 shutdown_connections 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1840543629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240d4590 0x7f65240d3530 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 -- 172.21.15.77:0/1840543629 >> 172.21.15.77:0/1840543629 conn(0x7f6524071a30 msgr2=0x7f6524071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 -- 172.21.15.77:0/1840543629 shutdown_connections 2024-01-15T14:35:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.805+0000 7f6529c4f700 1 -- 172.21.15.77:0/1840543629 wait complete. 2024-01-15T14:35:28.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.806+0000 7f6529c4f700 1 Processor -- start 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.806+0000 7f6529c4f700 1 -- start start 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6529c4f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 0x7f652415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6529c4f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f65240d4590 0x7f652415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6529c4f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f652415e110 0x7f65240b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6529c4f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652415fcd0 con 0x7f65240cf110 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6529c4f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652415ffb0 con 0x7f65240d4590 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6529c4f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6524160290 con 0x7f652415e110 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 0x7f652415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 0x7f652415a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40722/0 (socket says 172.21.15.77:40722) 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 -- 172.21.15.77:0/1010349978 learned_addr learned my addr 172.21.15.77:0/1010349978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6522ffd700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f65240d4590 0x7f652415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6523fff700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f652415e110 0x7f65240b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 -- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f652415e110 msgr2=0x7f65240b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f652415e110 0x7f65240b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 -- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f65240d4590 msgr2=0x7f652415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:28.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f65240d4590 0x7f652415ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f65237fe700 1 -- 172.21.15.77:0/1010349978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c009b30 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6522ffd700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f65240d4590 0x7f652415ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.807+0000 7f6523fff700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f652415e110 0x7f65240b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.808+0000 7f65237fe700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 0x7f652415a520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f651400b4a0 tx=0x7f651400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.808+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514011830 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.808+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6514011990 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.808+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514011d40 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.808+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6524160570 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.808+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652415aa60 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.809+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6524069020 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.809+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f651401a400 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.810+0000 7f650bfff700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f651006b350 0x7f651006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.810+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6514089850 con 0x7f65240cf110 2024-01-15T14:35:28.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.810+0000 7f6522ffd700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f651006b350 0x7f651006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:28.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.810+0000 7f6522ffd700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f651006b350 0x7f651006d810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6518000f60 tx=0x7f6518007d00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:28.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:28.814+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6514054750 con 0x7f65240cf110 2024-01-15T14:35:29.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.004+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65240cc1d0 con 0x7f65240cf110 2024-01-15T14:35:29.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.004+0000 7f650bfff700 1 -- 172.21.15.77:0/1010349978 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6514015020 con 0x7f65240cf110 2024-01-15T14:35:29.005 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f651006b350 msgr2=0x7f651006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f651006b350 0x7f651006d810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6518000f60 tx=0x7f6518007d00 comp rx=0 tx=0).stop 2024-01-15T14:35:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 msgr2=0x7f652415a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 0x7f652415a520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f651400b4a0 tx=0x7f651400b860 comp rx=0 tx=0).stop 2024-01-15T14:35:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 shutdown_connections 2024-01-15T14:35:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f651006b350 0x7f651006d810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f652415e110 0x7f65240b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65240cf110 0x7f652415a520 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 --2- 172.21.15.77:0/1010349978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f65240d4590 0x7f652415ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 >> 172.21.15.77:0/1010349978 conn(0x7f6524071a30 msgr2=0x7f65240ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 shutdown_connections 2024-01-15T14:35:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.007+0000 7f6529c4f700 1 -- 172.21.15.77:0/1010349978 wait complete. 2024-01-15T14:35:29.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:29.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.142+0000 7f7eeee54700 1 Processor -- start 2024-01-15T14:35:29.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.142+0000 7f7eeee54700 1 -- start start 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.143+0000 7f7eeee54700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ee815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.143+0000 7f7eeee54700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee815c2a0 con 0x7f7ee806d5b0 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.143+0000 7f7eeee54700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80b4560 con 0x7f7ee80ce510 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.143+0000 7f7eeee54700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80b4840 con 0x7f7ee81552c0 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.143+0000 7f7eed3f1700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7ee806d5b0 0x7f7ee8071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34974/0 (socket says 172.21.15.77:34974) 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.143+0000 7f7eed3f1700 1 -- 172.21.15.77:0/3064446387 learned_addr learned my addr 172.21.15.77:0/3064446387 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:29.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216746004 0 0) 0x7f7ee815c2a0 con 0x7f7ee806d5b0 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ee815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed80035b0 con 0x7f7ee806d5b0 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369549572 0 0) 0x7f7ed80035b0 con 0x7f7ee806d5b0 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 msgr2=0x7f7ee815bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ee815bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 >> v1:172.21.15.159:6789/0 conn(0x7f7ee80ce510 legacy=0x7f7ee806cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80b4b20 con 0x7f7ee806d5b0 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f7ee00019f0 con 0x7f7ee806d5b0 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 >> v1:172.21.15.77:6789/0 conn(0x7f7ee806d5b0 legacy=0x7f7ee8071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ed80043e0 0x7f7ed80048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ed80051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80b4840 con 0x7f7ed8003a30 2024-01-15T14:35:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed800bac0 con 0x7f7ed80043e0 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed800bc80 con 0x7f7ee81552c0 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eed3f1700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ed80051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7ee7fff700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ed80043e0 0x7f7ed80048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 -- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 msgr2=0x7f7ed80051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ed80051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 -- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ed80043e0 msgr2=0x7f7ed80048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ed80043e0 0x7f7ed80048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 -- 172.21.15.77:0/3064446387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80035b0 con 0x7f7ed8003a30 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eed3f1700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee81552c0 0x7f7ed80051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.144+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009b60 tx=0x7f7ed8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ead49c2aaf5d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00e6c0 con 0x7f7ed8003a30 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc00e820 con 0x7f7ed8003a30 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7ee77fe700 1 -- 172.21.15.77:0/3064446387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00eb90 con 0x7f7ed8003a30 2024-01-15T14:35:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eeee54700 1 -- 172.21.15.77:0/3064446387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 msgr2=0x7f7ed8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eeee54700 1 --2- 172.21.15.77:0/3064446387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ed8003e10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009b60 tx=0x7f7ed8005880 comp rx=0 tx=0).stop 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eed3f1700 1 -- 172.21.15.77:0/3064446387 reap_dead start 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eeee54700 1 -- 172.21.15.77:0/3064446387 shutdown_connections 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eeee54700 1 -- 172.21.15.77:0/3064446387 >> 172.21.15.77:0/3064446387 conn(0x7f7ee806aff0 msgr2=0x7f7ee806b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eeee54700 1 -- 172.21.15.77:0/3064446387 shutdown_connections 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.145+0000 7f7eeee54700 1 -- 172.21.15.77:0/3064446387 wait complete. 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.146+0000 7f7eeee54700 1 Processor -- start 2024-01-15T14:35:29.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.146+0000 7f7eeee54700 1 -- start start 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eeee54700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ee813a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eeee54700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee813c8a0 0x7f7ee80b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eeee54700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ee813cd20 0x7f7ee80b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eeee54700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee813d870 con 0x7f7ed8003a30 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eeee54700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee813d2b0 con 0x7f7ee813cd20 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eeee54700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee813d590 con 0x7f7ee813c8a0 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eecbf0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ee813a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eecbf0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ee813a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40740/0 (socket says 172.21.15.77:40740) 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eecbf0700 1 -- 172.21.15.77:0/2859856623 learned_addr learned my addr 172.21.15.77:0/2859856623 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7ee7fff700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee813c8a0 0x7f7ee80b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.147+0000 7f7eed3f1700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ee813cd20 0x7f7ee80b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eecbf0700 1 -- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee813c8a0 msgr2=0x7f7ee80b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee813c8a0 0x7f7ee80b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eecbf0700 1 -- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ee813cd20 msgr2=0x7f7ee80b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ee813cd20 0x7f7ee80b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eecbf0700 1 -- 172.21.15.77:0/2859856623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee813a990 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eed3f1700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ee813cd20 0x7f7ee80b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eecbf0700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ee813a130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8005810 tx=0x7f7edc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00e6c0 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc003ab0 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc003e60 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee813d870 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.148+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee8149f90 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.149+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee813d6d0 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.149+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7edc013470 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.150+0000 7f7ee57fa700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7ecc06b350 0x7f7ecc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.150+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7edc08b3e0 con 0x7f7ed8003a30 2024-01-15T14:35:29.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.150+0000 7f7ee7fff700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7ecc06b350 0x7f7ecc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.150+0000 7f7ee7fff700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7ecc06b350 0x7f7ecc06d810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40013a0 tx=0x7f7ed4001330 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.154+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ee813d6d0 con 0x7f7ed8003a30 2024-01-15T14:35:29.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.350+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7ee813d6d0 con 0x7f7ed8003a30 2024-01-15T14:35:29.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.351+0000 7f7ee57fa700 1 -- 172.21.15.77:0/2859856623 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ee813d6d0 con 0x7f7ed8003a30 2024-01-15T14:35:29.352 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7ecc06b350 msgr2=0x7f7ecc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7ecc06b350 0x7f7ecc06d810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40013a0 tx=0x7f7ed4001330 comp rx=0 tx=0).stop 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 msgr2=0x7f7ee813a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ee813a130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8005810 tx=0x7f7edc0047c0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 shutdown_connections 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7ecc06b350 0x7f7ecc06d810 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7ee813cd20 0x7f7ee80b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed8003a30 0x7f7ee813a130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 --2- 172.21.15.77:0/2859856623 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7ee813c8a0 0x7f7ee80b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 >> 172.21.15.77:0/2859856623 conn(0x7f7ee806aff0 msgr2=0x7f7ee815cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 shutdown_connections 2024-01-15T14:35:29.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.353+0000 7f7eeee54700 1 -- 172.21.15.77:0/2859856623 wait complete. 2024-01-15T14:35:29.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.487+0000 7fe985917700 1 Processor -- start 2024-01-15T14:35:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.487+0000 7fe985917700 1 -- start start 2024-01-15T14:35:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe985917700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fe98015e1e0 0x7fe9800d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe985917700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 0x7fe9800b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe985917700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9800d20b0 con 0x7fe9800d3a00 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe985917700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9800d3150 con 0x7fe98015e1e0 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe985917700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98015e5c0 con 0x7fe9800d0f80 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97e7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe9800d3a00 0x7fe9800d0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43500/0 (socket says 172.21.15.77:43500) 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97e7fc700 1 -- 172.21.15.77:0/2190623620 learned_addr learned my addr 172.21.15.77:0/2190623620 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702301048 0 0) 0x7fe9800d20b0 con 0x7fe9800d3a00 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9680035b0 con 0x7fe9800d3a00 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97effd700 1 --2- 172.21.15.77:0/2190623620 >> v2:172.21.15.159:3300/0 conn(0x7fe98015e1e0 0x7fe9800d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97f7fe700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 0x7fe9800b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967503305 0 0) 0x7fe9680035b0 con 0x7fe9800d3a00 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 msgr2=0x7fe9800b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 0x7fe9800b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 >> v2:172.21.15.159:3300/0 conn(0x7fe98015e1e0 msgr2=0x7fe9800d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 --2- 172.21.15.77:0/2190623620 >> v2:172.21.15.159:3300/0 conn(0x7fe98015e1e0 0x7fe9800d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9800d2390 con 0x7fe9800d3a00 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fe974001670 con 0x7fe9800d3a00 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 >> v1:172.21.15.77:6789/0 conn(0x7fe9800d3a00 legacy=0x7fe9800d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe968003a30 0x7fe968003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe968004850 0x7fe968004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 0x7fe968005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800d3150 con 0x7fe968004850 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98015e5c0 con 0x7fe968003a30 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.488+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96800ba60 con 0x7fe9800d0f80 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe968004850 0x7fe968004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97effd700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe968003a30 0x7fe968003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97f7fe700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 0x7fe968005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 -- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 msgr2=0x7fe968005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe9800d0f80 0x7fe968005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 -- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe968003a30 msgr2=0x7fe968003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe968003a30 0x7fe968003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 -- 172.21.15.77:0/2190623620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680035b0 con 0x7fe968004850 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe968004850 0x7fe968004cd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe97400c240 tx=0x7fe968005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9dd3e05f0ee437 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974007c10 con 0x7fe968004850 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97400cdc0 con 0x7fe968004850 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.489+0000 7fe97dffb700 1 -- 172.21.15.77:0/2190623620 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974005ba0 con 0x7fe968004850 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe985917700 1 -- 172.21.15.77:0/2190623620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe968004850 msgr2=0x7fe968004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe985917700 1 --2- 172.21.15.77:0/2190623620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe968004850 0x7fe968004cd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe97400c240 tx=0x7fe968005880 comp rx=0 tx=0).stop 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe97f7fe700 1 -- 172.21.15.77:0/2190623620 reap_dead start 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe985917700 1 -- 172.21.15.77:0/2190623620 shutdown_connections 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe985917700 1 -- 172.21.15.77:0/2190623620 >> 172.21.15.77:0/2190623620 conn(0x7fe9800b5b50 msgr2=0x7fe9800b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe985917700 1 -- 172.21.15.77:0/2190623620 shutdown_connections 2024-01-15T14:35:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.490+0000 7fe985917700 1 -- 172.21.15.77:0/2190623620 wait complete. 2024-01-15T14:35:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.491+0000 7fe985917700 1 Processor -- start 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.491+0000 7fe985917700 1 -- start start 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe968004850 0x7fe9800caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 0x7fe98013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe980140140 0x7fe9801405f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe980140b30 con 0x7fe9801610d0 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9801e72f0 con 0x7fe980140140 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9801e75d0 con 0x7fe968004850 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 0x7fe98013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 0x7fe98013fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57540/0 (socket says 172.21.15.77:57540) 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 -- 172.21.15.77:0/3574525409 learned_addr learned my addr 172.21.15.77:0/3574525409 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97effd700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe968004850 0x7fe9800caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97f7fe700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe980140140 0x7fe9801405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 -- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe968004850 msgr2=0x7fe9800caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe968004850 0x7fe9800caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 -- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe980140140 msgr2=0x7fe9801405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe980140140 0x7fe9801405f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 -- 172.21.15.77:0/3574525409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9801e78b0 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe97e7fc700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 0x7fe98013fc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe968004790 tx=0x7fe974001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974004c80 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe974004e00 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.492+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9801e7b90 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.493+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974007310 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.493+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9800cb620 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.493+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9800751e0 con 0x7fe9801610d0 2024-01-15T14:35:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.494+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe974019070 con 0x7fe9801610d0 2024-01-15T14:35:29.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.494+0000 7fe967fff700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe96c06b350 0x7fe96c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.494+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe97408a880 con 0x7fe9801610d0 2024-01-15T14:35:29.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.494+0000 7fe97effd700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe96c06b350 0x7fe96c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.495+0000 7fe97effd700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe96c06b350 0x7fe96c06d810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe970009c60 tx=0x7fe970009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.499+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe974055700 con 0x7fe9801610d0 2024-01-15T14:35:29.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.692+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9800b55d0 con 0x7fe9801610d0 2024-01-15T14:35:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.693+0000 7fe967fff700 1 -- 172.21.15.77:0/3574525409 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fe974058d20 con 0x7fe9801610d0 2024-01-15T14:35:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:29.694 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe96c06b350 msgr2=0x7fe96c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe96c06b350 0x7fe96c06d810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe970009c60 tx=0x7fe970009210 comp rx=0 tx=0).stop 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 msgr2=0x7fe98013fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 0x7fe98013fc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe968004790 tx=0x7fe974001a00 comp rx=0 tx=0).stop 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 shutdown_connections 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe96c06b350 0x7fe96c06d810 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe968004850 0x7fe9800caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe980140140 0x7fe9801405f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 --2- 172.21.15.77:0/3574525409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9801610d0 0x7fe98013fc00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 >> 172.21.15.77:0/3574525409 conn(0x7fe9800b5b50 msgr2=0x7fe9800b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 shutdown_connections 2024-01-15T14:35:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.695+0000 7fe985917700 1 -- 172.21.15.77:0/3574525409 wait complete. 2024-01-15T14:35:29.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:29.709 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.832+0000 7fc310846700 1 Processor -- start 2024-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.832+0000 7fc310846700 1 -- start start 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc310846700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc30c15b240 0x7fc30c06e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc310846700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc30c06e9a0 0x7fc30c15b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc310846700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c06eeb0 con 0x7fc30c15bb60 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc310846700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc30c06f190 con 0x7fc30c15b240 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc310846700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc30c06f470 con 0x7fc30c06e9a0 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc30a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc30c15b240 0x7fc30c06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc30a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc30c15b240 0x7fc30c06e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57552/0 (socket says 172.21.15.77:57552) 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc30a59c700 1 -- 172.21.15.77:0/4171481433 learned_addr learned my addr 172.21.15.77:0/4171481433 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.833+0000 7fc309d9b700 1 --2- 172.21.15.77:0/4171481433 >> v2:172.21.15.159:3300/0 conn(0x7fc30c06e9a0 0x7fc30c15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30a59c700 1 -- 172.21.15.77:0/4171481433 >> v2:172.21.15.159:3300/0 conn(0x7fc30c06e9a0 msgr2=0x7fc30c15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30a59c700 1 --2- 172.21.15.77:0/4171481433 >> v2:172.21.15.159:3300/0 conn(0x7fc30c06e9a0 0x7fc30c15b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30a59c700 1 -- 172.21.15.77:0/4171481433 >> v1:172.21.15.159:6789/0 conn(0x7fc30c15bb60 legacy=0x7fc30c06b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30a59c700 1 -- 172.21.15.77:0/4171481433 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c06f750 con 0x7fc30c15b240 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30a59c700 1 --2- 172.21.15.77:0/4171481433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c06e460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc30c15e700 tx=0x7fc300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa9af8649df4fda server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30959a700 1 -- 172.21.15.77:0/4171481433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011830 con 0x7fc30c15b240 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30959a700 1 -- 172.21.15.77:0/4171481433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300011990 con 0x7fc30c15b240 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc30959a700 1 -- 172.21.15.77:0/4171481433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011d20 con 0x7fc30c15b240 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc310846700 1 -- 172.21.15.77:0/4171481433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 msgr2=0x7fc30c06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.834+0000 7fc310846700 1 --2- 172.21.15.77:0/4171481433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c06e460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc30c15e700 tx=0x7fc300009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.835+0000 7fc310846700 1 -- 172.21.15.77:0/4171481433 shutdown_connections 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.835+0000 7fc310846700 1 --2- 172.21.15.77:0/4171481433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c06e460 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.835+0000 7fc310846700 1 --2- 172.21.15.77:0/4171481433 >> v2:172.21.15.159:3300/0 conn(0x7fc30c06e9a0 0x7fc30c15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.835+0000 7fc310846700 1 -- 172.21.15.77:0/4171481433 >> 172.21.15.77:0/4171481433 conn(0x7fc30c0d0c00 msgr2=0x7fc30c0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.835+0000 7fc310846700 1 -- 172.21.15.77:0/4171481433 shutdown_connections 2024-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.835+0000 7fc310846700 1 -- 172.21.15.77:0/4171481433 wait complete. 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.836+0000 7fc310846700 1 Processor -- start 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.836+0000 7fc310846700 1 -- start start 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c06e9a0 0x7fc30c1514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c09d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15bb60 0x7fc30c1f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c151a30 con 0x7fc30c15b240 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c09da40 con 0x7fc30c15bb60 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c09ab30 con 0x7fc30c06e9a0 2024-01-15T14:35:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c09d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c09d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57560/0 (socket says 172.21.15.77:57560) 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 -- 172.21.15.77:0/3214362599 learned_addr learned my addr 172.21.15.77:0/3214362599 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc30a59c700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c06e9a0 0x7fc30c1514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc30ad9d700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15bb60 0x7fc30c1f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 -- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c06e9a0 msgr2=0x7fc30c1514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c06e9a0 0x7fc30c1514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 -- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15bb60 msgr2=0x7fc30c1f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15bb60 0x7fc30c1f04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 -- 172.21.15.77:0/3214362599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc300009550 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc309d9b700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c09d260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00e870 tx=0x7fc2fc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc009590 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc013680 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30c09d7a0 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0139f0 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.837+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30c09b0b0 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.838+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30c069020 con 0x7fc30c15b240 2024-01-15T14:35:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.839+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc2fc013b50 con 0x7fc30c15b240 2024-01-15T14:35:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.839+0000 7fc2faffd700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2ec06b350 0x7fc2ec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.839+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2fc089d80 con 0x7fc30c15b240 2024-01-15T14:35:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.839+0000 7fc30a59c700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2ec06b350 0x7fc2ec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.840+0000 7fc30a59c700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2ec06b350 0x7fc2ec06d810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc300009c10 tx=0x7fc300008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:29.844+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2fc054c80 con 0x7fc30c15b240 2024-01-15T14:35:30.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.037+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc30c06dde0 con 0x7fc30c15b240 2024-01-15T14:35:30.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.037+0000 7fc2faffd700 1 -- 172.21.15.77:0/3214362599 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2fc0170a0 con 0x7fc30c15b240 2024-01-15T14:35:30.038 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:30.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2ec06b350 msgr2=0x7fc2ec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2ec06b350 0x7fc2ec06d810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc300009c10 tx=0x7fc300008030 comp rx=0 tx=0).stop 2024-01-15T14:35:30.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 msgr2=0x7fc30c09d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c09d260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00e870 tx=0x7fc2fc00eb80 comp rx=0 tx=0).stop 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 shutdown_connections 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2ec06b350 0x7fc2ec06d810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc30c06e9a0 0x7fc30c1514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc30c15bb60 0x7fc30c1f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 --2- 172.21.15.77:0/3214362599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc30c15b240 0x7fc30c09d260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 >> 172.21.15.77:0/3214362599 conn(0x7fc30c0d0c00 msgr2=0x7fc30c06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 shutdown_connections 2024-01-15T14:35:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.039+0000 7fc310846700 1 -- 172.21.15.77:0/3214362599 wait complete. 2024-01-15T14:35:30.052 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:30.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.175+0000 7fe6096f5700 1 Processor -- start 2024-01-15T14:35:30.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.175+0000 7fe6096f5700 1 -- start start 2024-01-15T14:35:30.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe6096f5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe60415e110 0x7fe60406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe6096f5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040affe0 con 0x7fe60406e420 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe6096f5700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60406d980 con 0x7fe6040afb30 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe6096f5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6040b0260 con 0x7fe60415e110 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe602ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe60415e110 0x7fe60406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe6037fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe60406e420 0x7fe604073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43508/0 (socket says 172.21.15.77:43508) 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.176+0000 7fe6037fe700 1 -- 172.21.15.77:0/1580105597 learned_addr learned my addr 172.21.15.77:0/1580105597 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe602ffd700 1 -- 172.21.15.77:0/1580105597 >> v1:172.21.15.159:6789/0 conn(0x7fe6040afb30 legacy=0x7fe60406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe602ffd700 1 -- 172.21.15.77:0/1580105597 >> v1:172.21.15.77:6789/0 conn(0x7fe60406e420 legacy=0x7fe604073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe602ffd700 1 -- 172.21.15.77:0/1580105597 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040b0540 con 0x7fe60415e110 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe602ffd700 1 --2- 172.21.15.77:0/1580105597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60415e110 0x7fe60406f680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009840 tx=0x7fe5f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601e2c76b97b2897 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe601ffb700 1 -- 172.21.15.77:0/1580105597 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8011830 con 0x7fe60415e110 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe601ffb700 1 -- 172.21.15.77:0/1580105597 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f8011990 con 0x7fe60415e110 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe601ffb700 1 -- 172.21.15.77:0/1580105597 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8011d20 con 0x7fe60415e110 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe6096f5700 1 -- 172.21.15.77:0/1580105597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60415e110 msgr2=0x7fe60406f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe6096f5700 1 --2- 172.21.15.77:0/1580105597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60415e110 0x7fe60406f680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009840 tx=0x7fe5f8009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe6096f5700 1 -- 172.21.15.77:0/1580105597 shutdown_connections 2024-01-15T14:35:30.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.177+0000 7fe6096f5700 1 --2- 172.21.15.77:0/1580105597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60415e110 0x7fe60406f680 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.178+0000 7fe6096f5700 1 -- 172.21.15.77:0/1580105597 >> 172.21.15.77:0/1580105597 conn(0x7fe6040b3250 msgr2=0x7fe6040d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:30.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.178+0000 7fe6096f5700 1 -- 172.21.15.77:0/1580105597 shutdown_connections 2024-01-15T14:35:30.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.178+0000 7fe6096f5700 1 -- 172.21.15.77:0/1580105597 wait complete. 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.179+0000 7fe6096f5700 1 Processor -- start 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.179+0000 7fe6096f5700 1 -- start start 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 0x7fe604141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6040afb30 0x7fe604142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe60415e110 0x7fe604073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe604143b30 con 0x7fe60406e420 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe604142eb0 con 0x7fe60415e110 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe604143190 con 0x7fe6040afb30 2024-01-15T14:35:30.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 0x7fe604141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 0x7fe604141b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57568/0 (socket says 172.21.15.77:57568) 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 -- 172.21.15.77:0/2506772932 learned_addr learned my addr 172.21.15.77:0/2506772932 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 -- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6040afb30 msgr2=0x7fe604142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6040afb30 0x7fe604142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 -- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe60415e110 msgr2=0x7fe604073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6037fe700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe60415e110 0x7fe604073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe60415e110 0x7fe604073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 -- 172.21.15.77:0/2506772932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80094a0 con 0x7fe60406e420 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe602ffd700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 0x7fe604141b60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80090b0 tx=0x7fe5f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80038a0 con 0x7fe60406e420 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f801e070 con 0x7fe60406e420 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60406b880 con 0x7fe60406e420 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8003c60 con 0x7fe60406e420 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.180+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe604143750 con 0x7fe60406e420 2024-01-15T14:35:30.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.181+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe604069020 con 0x7fe60406e420 2024-01-15T14:35:30.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.182+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe5f802d450 con 0x7fe60406e420 2024-01-15T14:35:30.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.182+0000 7fe5eb7fe700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe5f006b350 0x7fe5f006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.182+0000 7fe6027fc700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe5f006b350 0x7fe5f006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.182+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5f808ac20 con 0x7fe60406e420 2024-01-15T14:35:30.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.183+0000 7fe6027fc700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe5f006b350 0x7fe5f006d810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6040687f0 tx=0x7fe5f4007a00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.187+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5f8055b20 con 0x7fe60406e420 2024-01-15T14:35:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.377+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe60415d9c0 con 0x7fe60406e420 2024-01-15T14:35:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.377+0000 7fe5eb7fe700 1 -- 172.21.15.77:0/2506772932 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe5f8015020 con 0x7fe60406e420 2024-01-15T14:35:30.379 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:30.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe5f006b350 msgr2=0x7fe5f006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe5f006b350 0x7fe5f006d810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6040687f0 tx=0x7fe5f4007a00 comp rx=0 tx=0).stop 2024-01-15T14:35:30.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 msgr2=0x7fe604141b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 0x7fe604141b60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80090b0 tx=0x7fe5f8004750 comp rx=0 tx=0).stop 2024-01-15T14:35:30.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 shutdown_connections 2024-01-15T14:35:30.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe5f006b350 0x7fe5f006d810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe60415e110 0x7fe604073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60406e420 0x7fe604141b60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 --2- 172.21.15.77:0/2506772932 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe6040afb30 0x7fe604142460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 >> 172.21.15.77:0/2506772932 conn(0x7fe6040b3250 msgr2=0x7fe6040d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:30.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 shutdown_connections 2024-01-15T14:35:30.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.380+0000 7fe6096f5700 1 -- 172.21.15.77:0/2506772932 wait complete. 2024-01-15T14:35:30.393 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:30.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.516+0000 7fb378e51700 1 Processor -- start 2024-01-15T14:35:30.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.516+0000 7fb378e51700 1 -- start start 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb378e51700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb37406c550 0x7fb37406f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb378e51700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406aff0 0x7fb37406b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb378e51700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37415e860 con 0x7fb37415e480 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb378e51700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3740cf490 con 0x7fb37406c550 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb378e51700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740701c0 con 0x7fb37406aff0 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb371d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb37406c550 0x7fb37406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb371d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb37406c550 0x7fb37406f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57580/0 (socket says 172.21.15.77:57580) 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.517+0000 7fb371d9b700 1 -- 172.21.15.77:0/2516619100 learned_addr learned my addr 172.21.15.77:0/2516619100 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb372d9d700 1 --2- 172.21.15.77:0/2516619100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406aff0 0x7fb37406b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb371d9b700 1 -- 172.21.15.77:0/2516619100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406aff0 msgr2=0x7fb37406b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb371d9b700 1 --2- 172.21.15.77:0/2516619100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406aff0 0x7fb37406b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb371d9b700 1 -- 172.21.15.77:0/2516619100 >> v1:172.21.15.159:6789/0 conn(0x7fb37415e480 legacy=0x7fb37406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb371d9b700 1 -- 172.21.15.77:0/2516619100 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37406cb40 con 0x7fb37406c550 2024-01-15T14:35:30.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb371d9b700 1 --2- 172.21.15.77:0/2516619100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37406c550 0x7fb37406f590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0098f0 tx=0x7fb35c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bdd4b03ab746c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb37159a700 1 -- 172.21.15.77:0/2516619100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00b8a0 con 0x7fb37406c550 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb37159a700 1 -- 172.21.15.77:0/2516619100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c00ba00 con 0x7fb37406c550 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb37159a700 1 -- 172.21.15.77:0/2516619100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00bd90 con 0x7fb37406c550 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb378e51700 1 -- 172.21.15.77:0/2516619100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37406c550 msgr2=0x7fb37406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.518+0000 7fb378e51700 1 --2- 172.21.15.77:0/2516619100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37406c550 0x7fb37406f590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0098f0 tx=0x7fb35c009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.519+0000 7fb378e51700 1 -- 172.21.15.77:0/2516619100 shutdown_connections 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.519+0000 7fb378e51700 1 --2- 172.21.15.77:0/2516619100 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406aff0 0x7fb37406b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.519+0000 7fb378e51700 1 --2- 172.21.15.77:0/2516619100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37406c550 0x7fb37406f590 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.519+0000 7fb378e51700 1 -- 172.21.15.77:0/2516619100 >> 172.21.15.77:0/2516619100 conn(0x7fb3740b37f0 msgr2=0x7fb3740b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.519+0000 7fb378e51700 1 -- 172.21.15.77:0/2516619100 shutdown_connections 2024-01-15T14:35:30.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.519+0000 7fb378e51700 1 -- 172.21.15.77:0/2516619100 wait complete. 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.520+0000 7fb378e51700 1 Processor -- start 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.520+0000 7fb378e51700 1 -- start start 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb37406aff0 0x7fb3741514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406c550 0x7fb3740cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 0x7fb3740b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740ca3e0 con 0x7fb37415e480 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740c9810 con 0x7fb37406aff0 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740c9ab0 con 0x7fb37406c550 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 0x7fb3740b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 0x7fb3740b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57588/0 (socket says 172.21.15.77:57588) 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 -- 172.21.15.77:0/1428635028 learned_addr learned my addr 172.21.15.77:0/1428635028 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb371d9b700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406c550 0x7fb3740cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb37259c700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb37406aff0 0x7fb3741514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 -- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406c550 msgr2=0x7fb3740cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406c550 0x7fb3740cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 -- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb37406aff0 msgr2=0x7fb3741514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb37406aff0 0x7fb3741514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 -- 172.21.15.77:0/1428635028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c009550 con 0x7fb37415e480 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb371d9b700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406c550 0x7fb3740cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb37259c700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb37406aff0 0x7fb3741514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb372d9d700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 0x7fb3740b48e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00e810 tx=0x7fb36c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c013610 con 0x7fb37415e480 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c013770 con 0x7fb37415e480 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c013b20 con 0x7fb37415e480 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3740c9d90 con 0x7fb37415e480 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.521+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3740b6d30 con 0x7fb37415e480 2024-01-15T14:35:30.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.522+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3740ca550 con 0x7fb37415e480 2024-01-15T14:35:30.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.524+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb36c013c80 con 0x7fb37415e480 2024-01-15T14:35:30.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.524+0000 7fb36affd700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb35406b350 0x7fb35406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.524+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb36c089c10 con 0x7fb37415e480 2024-01-15T14:35:30.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.524+0000 7fb37259c700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb35406b350 0x7fb35406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.524+0000 7fb37259c700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb35406b350 0x7fb35406d810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb3640012e0 tx=0x7fb364009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.528+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb36c054b60 con 0x7fb37415e480 2024-01-15T14:35:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.719+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb374069020 con 0x7fb37415e480 2024-01-15T14:35:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.719+0000 7fb36affd700 1 -- 172.21.15.77:0/1428635028 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb36c021020 con 0x7fb37415e480 2024-01-15T14:35:30.720 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.721+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb35406b350 msgr2=0x7fb35406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.721+0000 7fb378e51700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb35406b350 0x7fb35406d810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb3640012e0 tx=0x7fb364009040 comp rx=0 tx=0).stop 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.721+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 msgr2=0x7fb3740b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.721+0000 7fb378e51700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 0x7fb3740b48e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00e810 tx=0x7fb36c00ebd0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 shutdown_connections 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb35406b350 0x7fb35406d810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb37415e480 0x7fb3740b48e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb37406aff0 0x7fb3741514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 --2- 172.21.15.77:0/1428635028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb37406c550 0x7fb3740cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 >> 172.21.15.77:0/1428635028 conn(0x7fb3740b37f0 msgr2=0x7fb37415b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 shutdown_connections 2024-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.722+0000 7fb378e51700 1 -- 172.21.15.77:0/1428635028 wait complete. 2024-01-15T14:35:30.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.855+0000 7f147ed62700 1 Processor -- start 2024-01-15T14:35:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.856+0000 7f147ed62700 1 -- start start 2024-01-15T14:35:30.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.856+0000 7f147ed62700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1478070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.856+0000 7f147ed62700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f147815b140 0x7f1478071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.856+0000 7f147ed62700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14780b3000 con 0x7f1478158770 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.856+0000 7f147ed62700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14780b32e0 con 0x7f147815b140 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.856+0000 7f147ed62700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b35c0 con 0x7f147806eb70 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f147cafe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1478158770 0x7f147806e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43510/0 (socket says 172.21.15.77:43510) 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f147cafe700 1 -- 172.21.15.77:0/1577039147 learned_addr learned my addr 172.21.15.77:0/1577039147 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f1477fff700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1478070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f147d2ff700 1 --2- 172.21.15.77:0/1577039147 >> v2:172.21.15.159:3300/0 conn(0x7f147815b140 0x7f1478071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287129561 0 0) 0x7f14780b3000 con 0x7f1478158770 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14680035b0 con 0x7f1478158770 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1970668052 0 0) 0x7f14680035b0 con 0x7f1478158770 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 msgr2=0x7f1478070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1478070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 >> v2:172.21.15.159:3300/0 conn(0x7f147815b140 msgr2=0x7f1478071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 --2- 172.21.15.77:0/1577039147 >> v2:172.21.15.159:3300/0 conn(0x7f147815b140 0x7f1478071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14780b38a0 con 0x7f1478158770 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f146c001670 con 0x7f1478158770 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 >> v1:172.21.15.77:6789/0 conn(0x7f1478158770 legacy=0x7f147806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1468003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1468003f80 0x7f1468004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1468004d90 0x7f1468005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b32e0 con 0x7f1468004d90 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b35c0 con 0x7f1468003f80 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146800ba10 con 0x7f147806eb70 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f147d2ff700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1468004d90 0x7f1468005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f1477fff700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1468003f80 0x7f1468004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f147cafe700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1468003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.857+0000 7f147d2ff700 1 -- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 msgr2=0x7f1468003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147d2ff700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1468003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147d2ff700 1 -- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1468003f80 msgr2=0x7f1468004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:30.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147d2ff700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1468003f80 0x7f1468004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147d2ff700 1 -- 172.21.15.77:0/1577039147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680035b0 con 0x7f1468004d90 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147cafe700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f147806eb70 0x7f1468003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147d2ff700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1468004d90 0x7f1468005210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1470009fc0 tx=0x7f1468005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b183a849b0b1fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147000ebd0 con 0x7f1468004d90 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147000f040 con 0x7f1468004d90 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f14777fe700 1 -- 172.21.15.77:0/1577039147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470014480 con 0x7f1468004d90 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147ed62700 1 -- 172.21.15.77:0/1577039147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1468004d90 msgr2=0x7f1468005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147ed62700 1 --2- 172.21.15.77:0/1577039147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1468004d90 0x7f1468005210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1470009fc0 tx=0x7f1468005880 comp rx=0 tx=0).stop 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147d2ff700 1 -- 172.21.15.77:0/1577039147 reap_dead start 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147ed62700 1 -- 172.21.15.77:0/1577039147 shutdown_connections 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147ed62700 1 -- 172.21.15.77:0/1577039147 >> 172.21.15.77:0/1577039147 conn(0x7f14780b1740 msgr2=0x7f14780b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147ed62700 1 -- 172.21.15.77:0/1577039147 shutdown_connections 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.858+0000 7f147ed62700 1 -- 172.21.15.77:0/1577039147 wait complete. 2024-01-15T14:35:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 Processor -- start 2024-01-15T14:35:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 -- start start 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14781f1da0 0x7f14780a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 0x7f14780c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14780c7590 0x7f147814a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781f2180 con 0x7f14780a99a0 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780a9e20 con 0x7f14780c7590 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.860+0000 7f147ed62700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14780c7db0 con 0x7f14781f1da0 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 0x7f14780c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 0x7f14780c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57612/0 (socket says 172.21.15.77:57612) 2024-01-15T14:35:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 -- 172.21.15.77:0/2586946550 learned_addr learned my addr 172.21.15.77:0/2586946550 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 -- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14781f1da0 msgr2=0x7f14780a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f147cafe700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14781f1da0 0x7f14780a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14781f1da0 0x7f14780a9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f147d2ff700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14780c7590 0x7f147814a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 -- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14780c7590 msgr2=0x7f147814a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14780c7590 0x7f147814a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 -- 172.21.15.77:0/2586946550 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478148040 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f1477fff700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 0x7f14780c7050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f14640119a0 tx=0x7f1464011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400f940 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146400faa0 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400fe30 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1478148320 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.861+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781488e0 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.862+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14780751e0 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.863+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1464014440 con 0x7f14780a99a0 2024-01-15T14:35:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.863+0000 7f14757fa700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f145c06b350 0x7f145c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.863+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f146408a2e0 con 0x7f14780a99a0 2024-01-15T14:35:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.863+0000 7f147cafe700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f145c06b350 0x7f145c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.863+0000 7f147cafe700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f145c06b350 0x7f145c06d810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f146c001eb0 tx=0x7f146c001b10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:30.868+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1464055200 con 0x7f14780a99a0 2024-01-15T14:35:31.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.059+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f147806ceb0 con 0x7f14780a99a0 2024-01-15T14:35:31.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.060+0000 7f14757fa700 1 -- 172.21.15.77:0/2586946550 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1464058820 con 0x7f14780a99a0 2024-01-15T14:35:31.061 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:31.062 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f145c06b350 msgr2=0x7f145c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f145c06b350 0x7f145c06d810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f146c001eb0 tx=0x7f146c001b10 comp rx=0 tx=0).stop 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 msgr2=0x7f14780c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 0x7f14780c7050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f14640119a0 tx=0x7f1464011d60 comp rx=0 tx=0).stop 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 shutdown_connections 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f145c06b350 0x7f145c06d810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f14781f1da0 0x7f14780a9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f14780c7590 0x7f147814a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 --2- 172.21.15.77:0/2586946550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14780a99a0 0x7f14780c7050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.062+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 >> 172.21.15.77:0/2586946550 conn(0x7f14780b1740 msgr2=0x7f1478072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:31.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.063+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 shutdown_connections 2024-01-15T14:35:31.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.063+0000 7f147ed62700 1 -- 172.21.15.77:0/2586946550 wait complete. 2024-01-15T14:35:31.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:31.076 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.200+0000 7fc7d5546700 1 Processor -- start 2024-01-15T14:35:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.200+0000 7fc7d5546700 1 -- start start 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7d5546700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc7d006f580 0x7fc7d015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7d5546700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d00b5600 con 0x7fc7d015ec50 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7d5546700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d006fa10 con 0x7fc7d0158600 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7d5546700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d015f0c0 con 0x7fc7d006f580 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7ce7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc7d006f580 0x7fc7d015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7cf7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc7d015ec50 0x7fc7d006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43526/0 (socket says 172.21.15.77:43526) 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.201+0000 7fc7cf7fe700 1 -- 172.21.15.77:0/4091082780 learned_addr learned my addr 172.21.15.77:0/4091082780 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7ce7fc700 1 -- 172.21.15.77:0/4091082780 >> v1:172.21.15.159:6789/0 conn(0x7fc7d0158600 legacy=0x7fc7d00b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7ce7fc700 1 -- 172.21.15.77:0/4091082780 >> v1:172.21.15.77:6789/0 conn(0x7fc7d015ec50 legacy=0x7fc7d006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7ce7fc700 1 -- 172.21.15.77:0/4091082780 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d006e4e0 con 0x7fc7d006f580 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7cdffb700 1 -- 172.21.15.77:0/4091082780 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165645625 0 0) 0x7fc7d00b5600 con 0x7fc7d015ec50 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7ce7fc700 1 --2- 172.21.15.77:0/4091082780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d006f580 0x7fc7d015e710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009870 tx=0x7fc7c0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f34a8f9f7b127ed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7cdffb700 1 -- 172.21.15.77:0/4091082780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0011830 con 0x7fc7d006f580 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7cdffb700 1 -- 172.21.15.77:0/4091082780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c0011990 con 0x7fc7d006f580 2024-01-15T14:35:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.202+0000 7fc7cdffb700 1 -- 172.21.15.77:0/4091082780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0011d20 con 0x7fc7d006f580 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 -- 172.21.15.77:0/4091082780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d006f580 msgr2=0x7fc7d015e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 --2- 172.21.15.77:0/4091082780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d006f580 0x7fc7d015e710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009870 tx=0x7fc7c0009b80 comp rx=0 tx=0).stop 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 -- 172.21.15.77:0/4091082780 shutdown_connections 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 --2- 172.21.15.77:0/4091082780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d006f580 0x7fc7d015e710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 -- 172.21.15.77:0/4091082780 >> 172.21.15.77:0/4091082780 conn(0x7fc7d00b3250 msgr2=0x7fc7d00d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 -- 172.21.15.77:0/4091082780 shutdown_connections 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.203+0000 7fc7d5546700 1 -- 172.21.15.77:0/4091082780 wait complete. 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.204+0000 7fc7d5546700 1 Processor -- start 2024-01-15T14:35:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.204+0000 7fc7d5546700 1 -- start start 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7d006f580 0x7fc7d0156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7d0158600 0x7fc7d014d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 0x7fc7d01f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00b0720 con 0x7fc7d015ec50 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00b0a00 con 0x7fc7d006f580 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00ae540 con 0x7fc7d0158600 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 0x7fc7d01f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 0x7fc7d01f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57638/0 (socket says 172.21.15.77:57638) 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 -- 172.21.15.77:0/1662952086 learned_addr learned my addr 172.21.15.77:0/1662952086 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7ce7fc700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7d0158600 0x7fc7d014d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7ceffd700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7d006f580 0x7fc7d0156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 -- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7d0158600 msgr2=0x7fc7d014d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7d0158600 0x7fc7d014d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 -- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7d006f580 msgr2=0x7fc7d0156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7d006f580 0x7fc7d0156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 -- 172.21.15.77:0/1662952086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00094a0 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7ceffd700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7d006f580 0x7fc7d0156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7cf7fe700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 0x7fc7d01f1930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800cba0 tx=0x7fc7c800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c800ab80 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c800ad00 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d00ae820 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8007310 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.205+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d00b0ce0 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.206+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d00625f0 con 0x7fc7d015ec50 2024-01-15T14:35:31.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.207+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc7c8007470 con 0x7fc7d015ec50 2024-01-15T14:35:31.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.207+0000 7fc7b77fe700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc7bc06b350 0x7fc7bc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.207+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7c80899c0 con 0x7fc7d015ec50 2024-01-15T14:35:31.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.207+0000 7fc7ceffd700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc7bc06b350 0x7fc7bc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.208+0000 7fc7ceffd700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc7bc06b350 0x7fc7bc06d810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00afc40 tx=0x7fc7c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.212+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7c80548c0 con 0x7fc7d015ec50 2024-01-15T14:35:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.402+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7d00cd440 con 0x7fc7d015ec50 2024-01-15T14:35:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.402+0000 7fc7b77fe700 1 -- 172.21.15.77:0/1662952086 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7c8018020 con 0x7fc7d015ec50 2024-01-15T14:35:31.404 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc7bc06b350 msgr2=0x7fc7bc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc7bc06b350 0x7fc7bc06d810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00afc40 tx=0x7fc7c4009040 comp rx=0 tx=0).stop 2024-01-15T14:35:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 msgr2=0x7fc7d01f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 0x7fc7d01f1930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800cba0 tx=0x7fc7c800cf60 comp rx=0 tx=0).stop 2024-01-15T14:35:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 shutdown_connections 2024-01-15T14:35:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc7bc06b350 0x7fc7bc06d810 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d015ec50 0x7fc7d01f1930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7d006f580 0x7fc7d0156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 --2- 172.21.15.77:0/1662952086 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc7d0158600 0x7fc7d014d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 >> 172.21.15.77:0/1662952086 conn(0x7fc7d00b3250 msgr2=0x7fc7d015b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 shutdown_connections 2024-01-15T14:35:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.405+0000 7fc7d5546700 1 -- 172.21.15.77:0/1662952086 wait complete. 2024-01-15T14:35:31.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:31.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.542+0000 7fa12dc3c700 1 Processor -- start 2024-01-15T14:35:31.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.542+0000 7fa12dc3c700 1 -- start start 2024-01-15T14:35:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa12dc3c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa1280afb80 0x7fa128073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa12dc3c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12815e4a0 con 0x7fa12815e0c0 2024-01-15T14:35:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa12dc3c700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12806aff0 con 0x7fa1280744d0 2024-01-15T14:35:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa12dc3c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1280b0010 con 0x7fa1280afb80 2024-01-15T14:35:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa126ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa1280afb80 0x7fa128073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa1277fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa12815e0c0 0x7fa1280b3bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43538/0 (socket says 172.21.15.77:43538) 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa1277fe700 1 -- 172.21.15.77:0/1060481331 learned_addr learned my addr 172.21.15.77:0/1060481331 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa126ffd700 1 -- 172.21.15.77:0/1060481331 >> v1:172.21.15.159:6789/0 conn(0x7fa1280744d0 legacy=0x7fa1280add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa126ffd700 1 -- 172.21.15.77:0/1060481331 >> v1:172.21.15.77:6789/0 conn(0x7fa12815e0c0 legacy=0x7fa1280b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa126ffd700 1 -- 172.21.15.77:0/1060481331 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12806b2d0 con 0x7fa1280afb80 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.543+0000 7fa1267fc700 1 -- 172.21.15.77:0/1060481331 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959130280 0 0) 0x7fa12815e4a0 con 0x7fa12815e0c0 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa126ffd700 1 --2- 172.21.15.77:0/1060481331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280afb80 0x7fa128073f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa118009870 tx=0x7fa118009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1848ade235fd8589 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa1267fc700 1 -- 172.21.15.77:0/1060481331 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118011830 con 0x7fa1280afb80 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa1267fc700 1 -- 172.21.15.77:0/1060481331 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa118011990 con 0x7fa1280afb80 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa1267fc700 1 -- 172.21.15.77:0/1060481331 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118011d20 con 0x7fa1280afb80 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 -- 172.21.15.77:0/1060481331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280afb80 msgr2=0x7fa128073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/1060481331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280afb80 0x7fa128073f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa118009870 tx=0x7fa118009b80 comp rx=0 tx=0).stop 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 -- 172.21.15.77:0/1060481331 shutdown_connections 2024-01-15T14:35:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/1060481331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280afb80 0x7fa128073f90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 -- 172.21.15.77:0/1060481331 >> 172.21.15.77:0/1060481331 conn(0x7fa1280d0c00 msgr2=0x7fa1280d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 -- 172.21.15.77:0/1060481331 shutdown_connections 2024-01-15T14:35:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.544+0000 7fa12dc3c700 1 -- 172.21.15.77:0/1060481331 wait complete. 2024-01-15T14:35:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.546+0000 7fa12dc3c700 1 Processor -- start 2024-01-15T14:35:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.546+0000 7fa12dc3c700 1 -- start start 2024-01-15T14:35:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.546+0000 7fa12dc3c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 0x7fa1281593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.546+0000 7fa12dc3c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280afb80 0x7fa1281450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.546+0000 7fa12dc3c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12815e0c0 0x7fa1280cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.546+0000 7fa12dc3c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cdb10 con 0x7fa1280744d0 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa12dc3c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cf690 con 0x7fa12815e0c0 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa12dc3c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cd170 con 0x7fa1280afb80 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 0x7fa1281593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 0x7fa1281593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57664/0 (socket says 172.21.15.77:57664) 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 -- 172.21.15.77:0/2702480935 learned_addr learned my addr 172.21.15.77:0/2702480935 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa127fff700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12815e0c0 0x7fa1280cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa126ffd700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280afb80 0x7fa1281450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 -- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280afb80 msgr2=0x7fa1281450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280afb80 0x7fa1281450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 -- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12815e0c0 msgr2=0x7fa1280cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12815e0c0 0x7fa1280cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 -- 172.21.15.77:0/2702480935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1180094a0 con 0x7fa1280744d0 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa126ffd700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280afb80 0x7fa1281450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa1277fe700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 0x7fa1281593f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa11000c520 tx=0x7fa11000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110004c80 con 0x7fa1280744d0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110004e20 con 0x7fa1280744d0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1280cd450 con 0x7fa1280744d0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110007310 con 0x7fa1280744d0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.547+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa128147e70 con 0x7fa1280744d0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.548+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1280cdc50 con 0x7fa1280744d0 2024-01-15T14:35:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.549+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa110007470 con 0x7fa1280744d0 2024-01-15T14:35:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.549+0000 7fa11ffff700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa10806b350 0x7fa10806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.549+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa110089890 con 0x7fa1280744d0 2024-01-15T14:35:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.549+0000 7fa126ffd700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa10806b350 0x7fa10806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.549+0000 7fa126ffd700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa10806b350 0x7fa10806d810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa128154bd0 tx=0x7fa118008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.554+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1280cdc50 con 0x7fa1280744d0 2024-01-15T14:35:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.743+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1280cdc50 con 0x7fa1280744d0 2024-01-15T14:35:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.743+0000 7fa11ffff700 1 -- 172.21.15.77:0/2702480935 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1280cdc50 con 0x7fa1280744d0 2024-01-15T14:35:31.744 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa10806b350 msgr2=0x7fa10806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa10806b350 0x7fa10806d810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa128154bd0 tx=0x7fa118008030 comp rx=0 tx=0).stop 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 msgr2=0x7fa1281593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 0x7fa1281593f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa11000c520 tx=0x7fa11000c8e0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 shutdown_connections 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa10806b350 0x7fa10806d810 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa12815e0c0 0x7fa1280cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1280744d0 0x7fa1281593f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 --2- 172.21.15.77:0/2702480935 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa1280afb80 0x7fa1281450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 >> 172.21.15.77:0/2702480935 conn(0x7fa1280d0c00 msgr2=0x7fa1280ae6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 shutdown_connections 2024-01-15T14:35:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.745+0000 7fa12dc3c700 1 -- 172.21.15.77:0/2702480935 wait complete. 2024-01-15T14:35:31.758 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:31.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.881+0000 7ffa38cb8700 1 Processor -- start 2024-01-15T14:35:31.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.882+0000 7ffa38cb8700 1 -- start start 2024-01-15T14:35:31.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.882+0000 7ffa38cb8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa340b53d0 0x7ffa3415b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.882+0000 7ffa38cb8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.882+0000 7ffa38cb8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa340ce870 con 0x7ffa34151d80 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.882+0000 7ffa38cb8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa34152160 con 0x7ffa340b53d0 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa38cb8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa340b3900 con 0x7ffa3406d3d0 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa31d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa340b53d0 0x7ffa3415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa3259c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffa34151d80 0x7ffa340b4cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43552/0 (socket says 172.21.15.77:43552) 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa3259c700 1 -- 172.21.15.77:0/4100567399 learned_addr learned my addr 172.21.15.77:0/4100567399 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa32d9d700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa31d9b700 1 -- 172.21.15.77:0/4100567399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 msgr2=0x7ffa340b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa31d9b700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa31d9b700 1 -- 172.21.15.77:0/4100567399 >> v1:172.21.15.77:6789/0 conn(0x7ffa34151d80 legacy=0x7ffa340b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa3159a700 1 -- 172.21.15.77:0/4100567399 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808109424 0 0) 0x7ffa340ce870 con 0x7ffa34151d80 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa31d9b700 1 -- 172.21.15.77:0/4100567399 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa340b43b0 con 0x7ffa340b53d0 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa32d9d700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa31d9b700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3415b130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400bae0 tx=0x7ffa2400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462e7c6c131c89ce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa3159a700 1 -- 172.21.15.77:0/4100567399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400dd30 con 0x7ffa340b53d0 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa3159a700 1 -- 172.21.15.77:0/4100567399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2400de90 con 0x7ffa340b53d0 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.883+0000 7ffa3159a700 1 -- 172.21.15.77:0/4100567399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24010690 con 0x7ffa340b53d0 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 -- 172.21.15.77:0/4100567399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 msgr2=0x7ffa3415b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3415b130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400bae0 tx=0x7ffa2400bdf0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 -- 172.21.15.77:0/4100567399 shutdown_connections 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/4100567399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3415b130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 -- 172.21.15.77:0/4100567399 >> 172.21.15.77:0/4100567399 conn(0x7ffa34074170 msgr2=0x7ffa3406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 -- 172.21.15.77:0/4100567399 shutdown_connections 2024-01-15T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.884+0000 7ffa38cb8700 1 -- 172.21.15.77:0/4100567399 wait complete. 2024-01-15T14:35:31.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.885+0000 7ffa38cb8700 1 Processor -- start 2024-01-15T14:35:31.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.885+0000 7ffa38cb8700 1 -- start start 2024-01-15T14:35:31.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa38cb8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa38cb8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3414a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa38cb8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa34151d80 0x7ffa3414acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa38cb8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3414a5b0 con 0x7ffa340b53d0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa38cb8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3414b390 con 0x7ffa34151d80 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa38cb8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3414b670 con 0x7ffa3406d3d0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3414a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3414a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57692/0 (socket says 172.21.15.77:57692) 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 -- 172.21.15.77:0/1367046234 learned_addr learned my addr 172.21.15.77:0/1367046234 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa3259c700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 -- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 msgr2=0x7ffa340b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa32d9d700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa34151d80 0x7ffa3414acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 -- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa34151d80 msgr2=0x7ffa3414acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa34151d80 0x7ffa3414acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.886+0000 7ffa31d9b700 1 -- 172.21.15.77:0/1367046234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2400b740 con 0x7ffa340b53d0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.887+0000 7ffa31d9b700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3414a7b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b2c0 tx=0x7ffa24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.887+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2401f070 con 0x7ffa340b53d0 2024-01-15T14:35:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.887+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24003bd0 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.887+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3414b950 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.887+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24024470 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.887+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa34138fb0 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.888+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa340753f0 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.888+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffa24019400 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.888+0000 7ffa2affd700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ffa1406b350 0x7ffa1406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.888+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa2408cfe0 con 0x7ffa340b53d0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.889+0000 7ffa3259c700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ffa1406b350 0x7ffa1406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.889+0000 7ffa3259c700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ffa1406b350 0x7ffa1406d810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c000f60 tx=0x7ffa1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:31.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:31.893+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa24057ee0 con 0x7ffa340b53d0 2024-01-15T14:35:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.081+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffa34069020 con 0x7ffa340b53d0 2024-01-15T14:35:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.082+0000 7ffa2affd700 1 -- 172.21.15.77:0/1367046234 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffa2405b500 con 0x7ffa340b53d0 2024-01-15T14:35:32.083 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:32.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ffa1406b350 msgr2=0x7ffa1406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ffa1406b350 0x7ffa1406d810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c000f60 tx=0x7ffa1c009040 comp rx=0 tx=0).stop 2024-01-15T14:35:32.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 msgr2=0x7ffa3414a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3414a7b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b2c0 tx=0x7ffa24004750 comp rx=0 tx=0).stop 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 shutdown_connections 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ffa1406b350 0x7ffa1406d810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ffa3406d3d0 0x7ffa340b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ffa34151d80 0x7ffa3414acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 --2- 172.21.15.77:0/1367046234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa340b53d0 0x7ffa3414a7b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 >> 172.21.15.77:0/1367046234 conn(0x7ffa34074170 msgr2=0x7ffa3415c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 shutdown_connections 2024-01-15T14:35:32.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.084+0000 7ffa38cb8700 1 -- 172.21.15.77:0/1367046234 wait complete. 2024-01-15T14:35:32.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:32.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.219+0000 7f8625d53700 1 Processor -- start 2024-01-15T14:35:32.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.219+0000 7f8625d53700 1 -- start start 2024-01-15T14:35:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.219+0000 7f8625d53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 0x7f86200d3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.219+0000 7f8625d53700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f86200d39d0 0x7f86200d4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f8625d53700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86200b80e0 con 0x7f86200b98c0 2024-01-15T14:35:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f8625d53700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862015e520 con 0x7f86200d39d0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f8625d53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86200d2930 con 0x7f862015e140 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861f7fe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 0x7f86200d3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861effd700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f86200d39d0 0x7f86200d4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861ffff700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f86200b98c0 0x7f86200b59b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:54168/0 (socket says 172.21.15.77:54168) 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861f7fe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 0x7f86200d3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:59810/0 (socket says 172.21.15.77:59810) 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861ffff700 1 -- 172.21.15.77:0/135991102 learned_addr learned my addr 172.21.15.77:0/135991102 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861effd700 1 -- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 msgr2=0x7f86200d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861effd700 1 --2- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 0x7f86200d3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861effd700 1 -- 172.21.15.77:0/135991102 >> v1:172.21.15.159:6789/0 conn(0x7f86200b98c0 legacy=0x7f86200b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861effd700 1 -- 172.21.15.77:0/135991102 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86200d2c10 con 0x7f86200d39d0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861e7fc700 1 -- 172.21.15.77:0/135991102 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099847303 0 0) 0x7f8618001490 con 0x7f86200b98c0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861f7fe700 1 --2- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 0x7f86200d3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.220+0000 7f861effd700 1 --2- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86200d39d0 0x7f86200d4400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8614009840 tx=0x7f8614009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc89e6c9912646c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f861e7fc700 1 -- 172.21.15.77:0/135991102 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614011830 con 0x7f86200d39d0 2024-01-15T14:35:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f861e7fc700 1 -- 172.21.15.77:0/135991102 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8614011990 con 0x7f86200d39d0 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f861e7fc700 1 -- 172.21.15.77:0/135991102 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614011d20 con 0x7f86200d39d0 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 -- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86200d39d0 msgr2=0x7f86200d4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 --2- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86200d39d0 0x7f86200d4400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8614009840 tx=0x7f8614009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 -- 172.21.15.77:0/135991102 shutdown_connections 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 --2- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f862015e140 0x7f86200d3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 --2- 172.21.15.77:0/135991102 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f86200d39d0 0x7f86200d4400 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 -- 172.21.15.77:0/135991102 >> 172.21.15.77:0/135991102 conn(0x7f862006b410 msgr2=0x7f862006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 -- 172.21.15.77:0/135991102 shutdown_connections 2024-01-15T14:35:32.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.221+0000 7f8625d53700 1 -- 172.21.15.77:0/135991102 wait complete. 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.222+0000 7f8625d53700 1 Processor -- start 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 -- start start 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 0x7f8620155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86200d39d0 0x7f86200b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f862015e140 0x7f86200cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8620155a90 con 0x7f86200b98c0 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86201f6fe0 con 0x7f862015e140 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f8625d53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86201f72c0 con 0x7f86200d39d0 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f861f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 0x7f8620155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f861f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 0x7f8620155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57698/0 (socket says 172.21.15.77:57698) 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.223+0000 7f861f7fe700 1 -- 172.21.15.77:0/1903345702 learned_addr learned my addr 172.21.15.77:0/1903345702 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861ffff700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f862015e140 0x7f86200cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861effd700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86200d39d0 0x7f86200b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861f7fe700 1 -- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86200d39d0 msgr2=0x7f86200b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861f7fe700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86200d39d0 0x7f86200b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861f7fe700 1 -- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f862015e140 msgr2=0x7f86200cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861f7fe700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f862015e140 0x7f86200cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861f7fe700 1 -- 172.21.15.77:0/1903345702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86100094a0 con 0x7f86200b98c0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861ffff700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f862015e140 0x7f86200cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f861f7fe700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 0x7f8620155550 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f861000ebe0 tx=0x7f8610009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861000c8b0 con 0x7f86200b98c0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861000ca10 con 0x7f86200b98c0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86140094a0 con 0x7f86200b98c0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861000cda0 con 0x7f86200b98c0 2024-01-15T14:35:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.224+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86201f7880 con 0x7f86200b98c0 2024-01-15T14:35:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.225+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8620068670 con 0x7f86200b98c0 2024-01-15T14:35:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.225+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8610013470 con 0x7f86200b98c0 2024-01-15T14:35:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.226+0000 7f8607fff700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f860c06b350 0x7f860c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.226+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8610089d30 con 0x7f86200b98c0 2024-01-15T14:35:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.226+0000 7f861effd700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f860c06b350 0x7f860c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.226+0000 7f861effd700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f860c06b350 0x7f860c06d810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8614000c00 tx=0x7f8614008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.230+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8610054b20 con 0x7f86200b98c0 2024-01-15T14:35:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.423+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86200751e0 con 0x7f86200b98c0 2024-01-15T14:35:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.424+0000 7f8607fff700 1 -- 172.21.15.77:0/1903345702 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f8610058140 con 0x7f86200b98c0 2024-01-15T14:35:32.425 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:32.426 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.426+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f860c06b350 msgr2=0x7f860c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.426+0000 7f8625d53700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f860c06b350 0x7f860c06d810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8614000c00 tx=0x7f8614008030 comp rx=0 tx=0).stop 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 msgr2=0x7f8620155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 0x7f8620155550 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f861000ebe0 tx=0x7f8610009e20 comp rx=0 tx=0).stop 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 shutdown_connections 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f860c06b350 0x7f860c06d810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f862015e140 0x7f86200cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86200b98c0 0x7f8620155550 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 --2- 172.21.15.77:0/1903345702 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f86200d39d0 0x7f86200b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 >> 172.21.15.77:0/1903345702 conn(0x7f862006b410 msgr2=0x7f86200d3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:32.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 shutdown_connections 2024-01-15T14:35:32.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.427+0000 7f8625d53700 1 -- 172.21.15.77:0/1903345702 wait complete. 2024-01-15T14:35:32.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:32.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:32.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.563+0000 7fea69639700 1 Processor -- start 2024-01-15T14:35:32.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.563+0000 7fea69639700 1 -- start start 2024-01-15T14:35:32.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea69639700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea640cf330 0x7fea640d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea69639700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea640d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea69639700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea64162af0 con 0x7fea64161510 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea69639700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea640d13e0 con 0x7fea640cf330 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea69639700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea64161f70 con 0x7fea640b98e0 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea640cf330 0x7fea640d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea640cf330 0x7fea640d2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57708/0 (socket says 172.21.15.77:57708) 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 -- 172.21.15.77:0/608562611 learned_addr learned my addr 172.21.15.77:0/608562611 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea637fe700 1 --2- 172.21.15.77:0/608562611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea640d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 -- 172.21.15.77:0/608562611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 msgr2=0x7fea640d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 --2- 172.21.15.77:0/608562611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea640d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 -- 172.21.15.77:0/608562611 >> v1:172.21.15.159:6789/0 conn(0x7fea64161510 legacy=0x7fea640cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 -- 172.21.15.77:0/608562611 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64162250 con 0x7fea640cf330 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea627fc700 1 --2- 172.21.15.77:0/608562611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea640cf330 0x7fea640d2700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fea540098f0 tx=0x7fea54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d229f3f03809aee server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea61ffb700 1 -- 172.21.15.77:0/608562611 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea54004230 con 0x7fea640cf330 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea61ffb700 1 -- 172.21.15.77:0/608562611 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea54004390 con 0x7fea640cf330 2024-01-15T14:35:32.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.564+0000 7fea61ffb700 1 -- 172.21.15.77:0/608562611 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea540036a0 con 0x7fea640cf330 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 -- 172.21.15.77:0/608562611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea640cf330 msgr2=0x7fea640d2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 --2- 172.21.15.77:0/608562611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea640cf330 0x7fea640d2700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fea540098f0 tx=0x7fea54009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 -- 172.21.15.77:0/608562611 shutdown_connections 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 --2- 172.21.15.77:0/608562611 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea640d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 --2- 172.21.15.77:0/608562611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea640cf330 0x7fea640d2700 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 -- 172.21.15.77:0/608562611 >> 172.21.15.77:0/608562611 conn(0x7fea6406b410 msgr2=0x7fea6406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 -- 172.21.15.77:0/608562611 shutdown_connections 2024-01-15T14:35:32.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.565+0000 7fea69639700 1 -- 172.21.15.77:0/608562611 wait complete. 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.566+0000 7fea69639700 1 Processor -- start 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.566+0000 7fea69639700 1 -- start start 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea69639700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea6415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea69639700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea640cf330 0x7fea640cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea69639700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 0x7fea640ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea69639700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6415e260 con 0x7fea64161510 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea69639700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6415d650 con 0x7fea640cf330 2024-01-15T14:35:32.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea69639700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6415d930 con 0x7fea640b98e0 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 0x7fea640ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 0x7fea640ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57720/0 (socket says 172.21.15.77:57720) 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea637fe700 1 -- 172.21.15.77:0/2634691293 learned_addr learned my addr 172.21.15.77:0/2634691293 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea627fc700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea640cf330 0x7fea640cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.567+0000 7fea62ffd700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea6415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea637fe700 1 -- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 msgr2=0x7fea6415d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea637fe700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea6415d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea637fe700 1 -- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea640cf330 msgr2=0x7fea640cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea637fe700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea640cf330 0x7fea640cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea637fe700 1 -- 172.21.15.77:0/2634691293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea54009550 con 0x7fea64161510 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea637fe700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 0x7fea640ca2d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fea5c007e40 tx=0x7fea5c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00ed30 con 0x7fea64161510 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c014420 con 0x7fea64161510 2024-01-15T14:35:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6415dc10 con 0x7fea64161510 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c0147d0 con 0x7fea64161510 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.568+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea640ca810 con 0x7fea64161510 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.569+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea64068450 con 0x7fea64161510 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.569+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fea5c014930 con 0x7fea64161510 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.570+0000 7fea4b7fe700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fea5006b350 0x7fea5006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.570+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea5c08a180 con 0x7fea64161510 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.570+0000 7fea62ffd700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fea5006b350 0x7fea5006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.570+0000 7fea62ffd700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fea5006b350 0x7fea5006d810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fea58009960 tx=0x7fea580090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.574+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea5c055080 con 0x7fea64161510 2024-01-15T14:35:32.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.761+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea64069020 con 0x7fea64161510 2024-01-15T14:35:32.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.762+0000 7fea4b7fe700 1 -- 172.21.15.77:0/2634691293 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea5c0586a0 con 0x7fea64161510 2024-01-15T14:35:32.763 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:32.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fea5006b350 msgr2=0x7fea5006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fea5006b350 0x7fea5006d810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fea58009960 tx=0x7fea580090e0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 msgr2=0x7fea640ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 0x7fea640ca2d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fea5c007e40 tx=0x7fea5c00e3c0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 shutdown_connections 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fea5006b350 0x7fea5006d810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fea640b98e0 0x7fea6415d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea64161510 0x7fea640ca2d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 --2- 172.21.15.77:0/2634691293 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fea640cf330 0x7fea640cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 >> 172.21.15.77:0/2634691293 conn(0x7fea6406b410 msgr2=0x7fea640d2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 shutdown_connections 2024-01-15T14:35:32.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.764+0000 7fea69639700 1 -- 172.21.15.77:0/2634691293 wait complete. 2024-01-15T14:35:32.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.900+0000 7fad2a7e5700 1 Processor -- start 2024-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.901+0000 7fad2a7e5700 1 -- start start 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.901+0000 7fad2a7e5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad2415ae00 0x7fad2406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.901+0000 7fad2a7e5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad240b3d60 con 0x7fad240add40 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.901+0000 7fad2a7e5700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad240b4040 con 0x7fad240d05b0 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.901+0000 7fad2a7e5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2415f8b0 con 0x7fad2415ae00 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad23fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad2415ae00 0x7fad2406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad237fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fad240add40 0x7fad240cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43560/0 (socket says 172.21.15.77:43560) 2024-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad237fe700 1 -- 172.21.15.77:0/2508384411 learned_addr learned my addr 172.21.15.77:0/2508384411 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad23fff700 1 -- 172.21.15.77:0/2508384411 >> v1:172.21.15.159:6789/0 conn(0x7fad240d05b0 legacy=0x7fad240b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad23fff700 1 -- 172.21.15.77:0/2508384411 >> v1:172.21.15.77:6789/0 conn(0x7fad240add40 legacy=0x7fad240cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad23fff700 1 -- 172.21.15.77:0/2508384411 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad240b5be0 con 0x7fad2415ae00 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad22ffd700 1 -- 172.21.15.77:0/2508384411 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259836713 0 0) 0x7fad240b3d60 con 0x7fad240add40 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad23fff700 1 --2- 172.21.15.77:0/2508384411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2415ae00 0x7fad2406dc60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009840 tx=0x7fad0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c48a8375b177265 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad22ffd700 1 -- 172.21.15.77:0/2508384411 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c011830 con 0x7fad2415ae00 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad22ffd700 1 -- 172.21.15.77:0/2508384411 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c011990 con 0x7fad2415ae00 2024-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.902+0000 7fad22ffd700 1 -- 172.21.15.77:0/2508384411 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c011d20 con 0x7fad2415ae00 2024-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 -- 172.21.15.77:0/2508384411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2415ae00 msgr2=0x7fad2406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/2508384411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2415ae00 0x7fad2406dc60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009840 tx=0x7fad0c009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 -- 172.21.15.77:0/2508384411 shutdown_connections 2024-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/2508384411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2415ae00 0x7fad2406dc60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 -- 172.21.15.77:0/2508384411 >> 172.21.15.77:0/2508384411 conn(0x7fad24072b20 msgr2=0x7fad2406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 -- 172.21.15.77:0/2508384411 shutdown_connections 2024-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.903+0000 7fad2a7e5700 1 -- 172.21.15.77:0/2508384411 wait complete. 2024-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.904+0000 7fad2a7e5700 1 Processor -- start 2024-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.904+0000 7fad2a7e5700 1 -- start start 2024-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad2a7e5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad240add40 0x7fad24152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad2a7e5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 0x7fad24159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad2a7e5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad2415ae00 0x7fad2406e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad2a7e5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2415d4c0 con 0x7fad240d05b0 2024-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad2a7e5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2415c8b0 con 0x7fad240add40 2024-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad2a7e5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2415cb90 con 0x7fad2415ae00 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 0x7fad24159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 0x7fad24159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57732/0 (socket says 172.21.15.77:57732) 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.905+0000 7fad237fe700 1 -- 172.21.15.77:0/1612749708 learned_addr learned my addr 172.21.15.77:0/1612749708 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad23fff700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad240add40 0x7fad24152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad237fe700 1 -- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad2415ae00 msgr2=0x7fad2406e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad28d82700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad2415ae00 0x7fad2406e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad237fe700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad2415ae00 0x7fad2406e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad237fe700 1 -- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad240add40 msgr2=0x7fad24152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad237fe700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad240add40 0x7fad24152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad237fe700 1 -- 172.21.15.77:0/1612749708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0094a0 con 0x7fad240d05b0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad23fff700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad240add40 0x7fad24152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad237fe700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 0x7fad24159b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c520 tx=0x7fad1400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14004c80 con 0x7fad240d05b0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad14004e00 con 0x7fad240d05b0 2024-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2415ce70 con 0x7fad240d05b0 2024-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14007310 con 0x7fad240d05b0 2024-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.906+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad24152670 con 0x7fad240d05b0 2024-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.907+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad24069020 con 0x7fad240d05b0 2024-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.907+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fad14007470 con 0x7fad240d05b0 2024-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.908+0000 7fad20ff9700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fad0406b350 0x7fad0406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.908+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad140899c0 con 0x7fad240d05b0 2024-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.908+0000 7fad23fff700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fad0406b350 0x7fad0406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.908+0000 7fad23fff700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fad0406b350 0x7fad0406d810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fad24152e00 tx=0x7fad0c01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:32.912+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad140547a0 con 0x7fad240d05b0 2024-01-15T14:35:33.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:35:33.089 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.104+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad2415d600 con 0x7fad240d05b0 2024-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.105+0000 7fad20ff9700 1 -- 172.21.15.77:0/1612749708 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad14057dc0 con 0x7fad240d05b0 2024-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fad0406b350 msgr2=0x7fad0406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fad0406b350 0x7fad0406d810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fad24152e00 tx=0x7fad0c01c040 comp rx=0 tx=0).stop 2024-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 msgr2=0x7fad24159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 0x7fad24159b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c520 tx=0x7fad1400c8e0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 shutdown_connections 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fad0406b350 0x7fad0406d810 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fad2415ae00 0x7fad2406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fad240add40 0x7fad24152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 --2- 172.21.15.77:0/1612749708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad240d05b0 0x7fad24159b50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 >> 172.21.15.77:0/1612749708 conn(0x7fad24072b20 msgr2=0x7fad240b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 shutdown_connections 2024-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.107+0000 7fad2a7e5700 1 -- 172.21.15.77:0/1612749708 wait complete. 2024-01-15T14:35:33.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:33.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.311+0000 7f5a712ce700 1 Processor -- start 2024-01-15T14:35:33.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.311+0000 7f5a712ce700 1 -- start start 2024-01-15T14:35:33.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a712ce700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5a6c161510 0x7f5a6c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a712ce700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c0b98e0 0x7f5a6c0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a712ce700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a6c0b59d0 0x7f5a6c0b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a712ce700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c162af0 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a712ce700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c0d13e0 con 0x7f5a6c161510 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a712ce700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c161f70 con 0x7f5a6c0b98e0 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a6c0b59d0 0x7f5a6c0b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a6c0b59d0 0x7f5a6c0b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57748/0 (socket says 172.21.15.77:57748) 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 -- 172.21.15.77:0/939887488 learned_addr learned my addr 172.21.15.77:0/939887488 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6a7fc700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c0b98e0 0x7f5a6c0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6affd700 1 --2- 172.21.15.77:0/939887488 >> v2:172.21.15.159:3300/0 conn(0x7f5a6c161510 0x7f5a6c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 -- 172.21.15.77:0/939887488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c0b98e0 msgr2=0x7f5a6c0cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:33.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c0b98e0 0x7f5a6c0cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 -- 172.21.15.77:0/939887488 >> v2:172.21.15.159:3300/0 conn(0x7f5a6c161510 msgr2=0x7f5a6c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 --2- 172.21.15.77:0/939887488 >> v2:172.21.15.159:3300/0 conn(0x7f5a6c161510 0x7f5a6c0cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6b7fe700 1 -- 172.21.15.77:0/939887488 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c162250 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6a7fc700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c0b98e0 0x7f5a6c0cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.312+0000 7f5a6affd700 1 --2- 172.21.15.77:0/939887488 >> v2:172.21.15.159:3300/0 conn(0x7f5a6c161510 0x7f5a6c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a6b7fe700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0b5e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0d49f0 tx=0x7f5a6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6faeee36ccad4db server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a69ffb700 1 -- 172.21.15.77:0/939887488 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400e070 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a69ffb700 1 -- 172.21.15.77:0/939887488 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6400b9c0 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a69ffb700 1 -- 172.21.15.77:0/939887488 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400bd70 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 -- 172.21.15.77:0/939887488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 msgr2=0x7f5a6c0b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0b5e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0d49f0 tx=0x7f5a6400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 -- 172.21.15.77:0/939887488 shutdown_connections 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 --2- 172.21.15.77:0/939887488 >> v2:172.21.15.159:3300/0 conn(0x7f5a6c161510 0x7f5a6c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0b5e60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 --2- 172.21.15.77:0/939887488 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c0b98e0 0x7f5a6c0cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 -- 172.21.15.77:0/939887488 >> 172.21.15.77:0/939887488 conn(0x7f5a6c06b410 msgr2=0x7f5a6c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 -- 172.21.15.77:0/939887488 shutdown_connections 2024-01-15T14:35:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.313+0000 7f5a712ce700 1 -- 172.21.15.77:0/939887488 wait complete. 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 Processor -- start 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 -- start start 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a6c0b98e0 0x7f5a6c1f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c161510 0x7f5a6c151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c15db30 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c15dd80 con 0x7f5a6c0b98e0 2024-01-15T14:35:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.315+0000 7f5a712ce700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c15e060 con 0x7f5a6c161510 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0acaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57754/0 (socket says 172.21.15.77:57754) 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 -- 172.21.15.77:0/3096059617 learned_addr learned my addr 172.21.15.77:0/3096059617 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6a7fc700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a6c0b98e0 0x7f5a6c1f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6b7fe700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c161510 0x7f5a6c151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 -- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c161510 msgr2=0x7f5a6c151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c161510 0x7f5a6c151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 -- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a6c0b98e0 msgr2=0x7f5a6c1f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a6c0b98e0 0x7f5a6c1f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 -- 172.21.15.77:0/3096059617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64009b30 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6b7fe700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c161510 0x7f5a6c151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6a7fc700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a6c0b98e0 0x7f5a6c1f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a6affd700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0acaf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000e810 tx=0x7f5a6000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000c020 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60004250 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c15e340 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a600036a0 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.316+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c0ad690 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.317+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c068450 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.318+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a60003800 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.318+0000 7f5a537fe700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5a5806b350 0x7f5a5806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.318+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a60089610 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.318+0000 7f5a6a7fc700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5a5806b350 0x7f5a5806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.318+0000 7f5a6a7fc700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5a5806b350 0x7f5a5806d810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c160890 tx=0x7f5a5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.322+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a60054480 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.512+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a6c069020 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.513+0000 7f5a537fe700 1 -- 172.21.15.77:0/3096059617 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a60057aa0 con 0x7f5a6c0b59d0 2024-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5a5806b350 msgr2=0x7f5a5806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5a5806b350 0x7f5a5806d810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c160890 tx=0x7f5a5c008040 comp rx=0 tx=0).stop 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 msgr2=0x7f5a6c0acaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0acaf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000e810 tx=0x7f5a6000ebd0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 shutdown_connections 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5a5806b350 0x7f5a5806d810 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5a6c161510 0x7f5a6c151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a6c0b59d0 0x7f5a6c0acaf0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 --2- 172.21.15.77:0/3096059617 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5a6c0b98e0 0x7f5a6c1f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 >> 172.21.15.77:0/3096059617 conn(0x7f5a6c06b410 msgr2=0x7f5a6c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 shutdown_connections 2024-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.515+0000 7f5a712ce700 1 -- 172.21.15.77:0/3096059617 wait complete. 2024-01-15T14:35:33.593 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-15T14:35:33.593 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T14:35:33.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:33.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.715+0000 7f1c3a294700 1 Processor -- start 2024-01-15T14:35:33.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.715+0000 7f1c3a294700 1 -- start start 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c3a294700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c34073f90 0x7f1c340ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c3a294700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1c3406ee50 0x7f1c34074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c3a294700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c3a294700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c34074e90 con 0x7f1c34073f90 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c3a294700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c340ce5d0 con 0x7f1c3406ee50 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c3a294700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c340748b0 con 0x7f1c3415c140 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c34073f90 0x7f1c340ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c34073f90 0x7f1c340ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57758/0 (socket says 172.21.15.77:57758) 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 -- 172.21.15.77:0/65436353 learned_addr learned my addr 172.21.15.77:0/65436353 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c337fe700 1 --2- 172.21.15.77:0/65436353 >> v2:172.21.15.159:3300/0 conn(0x7f1c3406ee50 0x7f1c34074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c38831700 1 --2- 172.21.15.77:0/65436353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 -- 172.21.15.77:0/65436353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 msgr2=0x7f1c3415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 --2- 172.21.15.77:0/65436353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 -- 172.21.15.77:0/65436353 >> v2:172.21.15.159:3300/0 conn(0x7f1c3406ee50 msgr2=0x7f1c34074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 --2- 172.21.15.77:0/65436353 >> v2:172.21.15.159:3300/0 conn(0x7f1c3406ee50 0x7f1c34074370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 -- 172.21.15.77:0/65436353 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3415c700 con 0x7f1c34073f90 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c33fff700 1 --2- 172.21.15.77:0/65436353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c340ce090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1c3406cfe0 tx=0x7f1c24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3c1b9eb7836a0f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c32ffd700 1 -- 172.21.15.77:0/65436353 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400b870 con 0x7f1c34073f90 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c32ffd700 1 -- 172.21.15.77:0/65436353 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2400b9d0 con 0x7f1c34073f90 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.716+0000 7f1c32ffd700 1 -- 172.21.15.77:0/65436353 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400bd60 con 0x7f1c34073f90 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 -- 172.21.15.77:0/65436353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 msgr2=0x7f1c340ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 --2- 172.21.15.77:0/65436353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c340ce090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1c3406cfe0 tx=0x7f1c24009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 -- 172.21.15.77:0/65436353 shutdown_connections 2024-01-15T14:35:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 --2- 172.21.15.77:0/65436353 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 --2- 172.21.15.77:0/65436353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c340ce090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 --2- 172.21.15.77:0/65436353 >> v2:172.21.15.159:3300/0 conn(0x7f1c3406ee50 0x7f1c34074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 -- 172.21.15.77:0/65436353 >> 172.21.15.77:0/65436353 conn(0x7f1c340cd570 msgr2=0x7f1c340cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:33.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 -- 172.21.15.77:0/65436353 shutdown_connections 2024-01-15T14:35:33.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.717+0000 7f1c3a294700 1 -- 172.21.15.77:0/65436353 wait complete. 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.718+0000 7f1c3a294700 1 Processor -- start 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 -- start start 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c3406ee50 0x7f1c341f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c3415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3415edd0 con 0x7f1c34073f90 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3415e810 con 0x7f1c3406ee50 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.719+0000 7f1c3a294700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3415eaf0 con 0x7f1c3415c140 2024-01-15T14:35:33.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c3415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c3415dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57772/0 (socket says 172.21.15.77:57772) 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 -- 172.21.15.77:0/124985701 learned_addr learned my addr 172.21.15.77:0/124985701 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c33fff700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c3406ee50 0x7f1c341f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c38831700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 -- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 msgr2=0x7f1c3415ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 -- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c3406ee50 msgr2=0x7f1c341f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c3406ee50 0x7f1c341f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 -- 172.21.15.77:0/124985701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24009550 con 0x7f1c34073f90 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c33fff700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c3406ee50 0x7f1c341f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c38831700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c337fe700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c3415dd90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800c7f0 tx=0x7f1c2800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28011800 con 0x7f1c34073f90 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c34144400 con 0x7f1c34073f90 2024-01-15T14:35:33.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c28011960 con 0x7f1c34073f90 2024-01-15T14:35:33.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28011d10 con 0x7f1c34073f90 2024-01-15T14:35:33.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.720+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c341f3aa0 con 0x7f1c34073f90 2024-01-15T14:35:33.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.721+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c34144920 con 0x7f1c34073f90 2024-01-15T14:35:33.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.722+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1c28018360 con 0x7f1c34073f90 2024-01-15T14:35:33.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.722+0000 7f1c30ff9700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1c2006b350 0x7f1c2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:33.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.722+0000 7f1c33fff700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1c2006b350 0x7f1c2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:33.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.722+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c28089b50 con 0x7f1c34073f90 2024-01-15T14:35:33.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.722+0000 7f1c33fff700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1c2006b350 0x7f1c2006d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1c3414a9e0 tx=0x7f1c24008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:33.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.727+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c34144920 con 0x7f1c34073f90 2024-01-15T14:35:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.918+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c34144920 con 0x7f1c34073f90 2024-01-15T14:35:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.919+0000 7f1c30ff9700 1 -- 172.21.15.77:0/124985701 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1c34144920 con 0x7f1c34073f90 2024-01-15T14:35:33.920 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:33.920 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1c2006b350 msgr2=0x7f1c2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1c2006b350 0x7f1c2006d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1c3414a9e0 tx=0x7f1c24008030 comp rx=0 tx=0).stop 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 msgr2=0x7f1c3415dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c3415dd90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800c7f0 tx=0x7f1c2800cbb0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 shutdown_connections 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1c2006b350 0x7f1c2006d810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1c3415c140 0x7f1c3415ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1c3406ee50 0x7f1c341f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 --2- 172.21.15.77:0/124985701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c34073f90 0x7f1c3415dd90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 >> 172.21.15.77:0/124985701 conn(0x7f1c340cd570 msgr2=0x7f1c340b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 shutdown_connections 2024-01-15T14:35:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:33.921+0000 7f1c3a294700 1 -- 172.21.15.77:0/124985701 wait complete. 2024-01-15T14:35:33.934 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:33.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:34.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.057+0000 7fa2fbf8d700 1 Processor -- start 2024-01-15T14:35:34.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.058+0000 7fa2fbf8d700 1 -- start start 2024-01-15T14:35:34.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fbf8d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 0x7fa2f40b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fbf8d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 0x7fa2f40ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fbf8d700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40b3860 con 0x7fa2f40cffc0 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fbf8d700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f40b2520 con 0x7fa2f40d03a0 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fbf8d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f40b2800 con 0x7fa2f415f8d0 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fa52a700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 0x7fa2f40ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 0x7fa2f40b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 0x7fa2f40b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:59862/0 (socket says 172.21.15.77:59862) 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fa52a700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 0x7fa2f40ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:43960/0 (socket says 172.21.15.77:43960) 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 -- 172.21.15.77:0/2896573248 learned_addr learned my addr 172.21.15.77:0/2896573248 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 -- 172.21.15.77:0/2896573248 >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 msgr2=0x7fa2f40ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 --2- 172.21.15.77:0/2896573248 >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 0x7fa2f40ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 -- 172.21.15.77:0/2896573248 >> v1:172.21.15.159:6789/0 conn(0x7fa2f40cffc0 legacy=0x7fa2f415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2f9528700 1 -- 172.21.15.77:0/2896573248 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f40b2ae0 con 0x7fa2f415f8d0 2024-01-15T14:35:34.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.059+0000 7fa2fa52a700 1 --2- 172.21.15.77:0/2896573248 >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 0x7fa2f40ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:34.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2f9528700 1 --2- 172.21.15.77:0/2896573248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 0x7fa2f40b3320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00098f0 tx=0x7fa2f0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a61e89d5a578b65 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2f8d27700 1 -- 172.21.15.77:0/2896573248 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0011830 con 0x7fa2f415f8d0 2024-01-15T14:35:34.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2f8d27700 1 -- 172.21.15.77:0/2896573248 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f0011990 con 0x7fa2f415f8d0 2024-01-15T14:35:34.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2f8d27700 1 -- 172.21.15.77:0/2896573248 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0011d20 con 0x7fa2f415f8d0 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/2896573248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 msgr2=0x7fa2f40b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/2896573248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 0x7fa2f40b3320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00098f0 tx=0x7fa2f0009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/2896573248 shutdown_connections 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/2896573248 >> v2:172.21.15.159:3300/0 conn(0x7fa2f40d03a0 0x7fa2f40ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/2896573248 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f415f8d0 0x7fa2f40b3320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/2896573248 >> 172.21.15.77:0/2896573248 conn(0x7fa2f40d1e50 msgr2=0x7fa2f40d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/2896573248 shutdown_connections 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.060+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/2896573248 wait complete. 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 Processor -- start 2024-01-15T14:35:34.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 -- start start 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f40cffc0 0x7fa2f40c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 0x7fa2f415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2f415f8d0 0x7fa2f40c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f406b590 con 0x7fa2f40d03a0 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f406b870 con 0x7fa2f415f8d0 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.062+0000 7fa2fbf8d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f406bb50 con 0x7fa2f40cffc0 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 0x7fa2f415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 0x7fa2f415b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57774/0 (socket says 172.21.15.77:57774) 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 -- 172.21.15.77:0/1585013387 learned_addr learned my addr 172.21.15.77:0/1585013387 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:34.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9d29700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f40cffc0 0x7fa2f40c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2fa52a700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2f415f8d0 0x7fa2f40c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 -- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f40cffc0 msgr2=0x7fa2f40c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f40cffc0 0x7fa2f40c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 -- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2f415f8d0 msgr2=0x7fa2f40c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2f415f8d0 0x7fa2f40c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 -- 172.21.15.77:0/1585013387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec009a70 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2f9528700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 0x7fa2f415b510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009c40 tx=0x7fa2f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0004510 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f001e070 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f0009550 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00038e0 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.063+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f406c110 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.064+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f406c300 con 0x7fa2f40d03a0 2024-01-15T14:35:34.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.064+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa2f0003a40 con 0x7fa2f40d03a0 2024-01-15T14:35:34.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.065+0000 7fa2ea7fc700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2d806b350 0x7fa2d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.065+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2f008a9f0 con 0x7fa2f40d03a0 2024-01-15T14:35:34.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.065+0000 7fa2f9d29700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2d806b350 0x7fa2d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.065+0000 7fa2f9d29700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2d806b350 0x7fa2d806d810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4007fd0 tx=0x7fa2e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.069+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2f0055870 con 0x7fa2f40d03a0 2024-01-15T14:35:34.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.261+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa2f4069020 con 0x7fa2f40d03a0 2024-01-15T14:35:34.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.262+0000 7fa2ea7fc700 1 -- 172.21.15.77:0/1585013387 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa2f0058e90 con 0x7fa2f40d03a0 2024-01-15T14:35:34.263 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2d806b350 msgr2=0x7fa2d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2d806b350 0x7fa2d806d810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4007fd0 tx=0x7fa2e4009040 comp rx=0 tx=0).stop 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 msgr2=0x7fa2f415b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 0x7fa2f415b510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009c40 tx=0x7fa2f0004750 comp rx=0 tx=0).stop 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 shutdown_connections 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2d806b350 0x7fa2d806d810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2f40cffc0 0x7fa2f40c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2f415f8d0 0x7fa2f40c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 --2- 172.21.15.77:0/1585013387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2f40d03a0 0x7fa2f415b510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 >> 172.21.15.77:0/1585013387 conn(0x7fa2f40d1e50 msgr2=0x7fa2f40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:34.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 shutdown_connections 2024-01-15T14:35:34.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.264+0000 7fa2fbf8d700 1 -- 172.21.15.77:0/1585013387 wait complete. 2024-01-15T14:35:34.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:34.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.399+0000 7f348de65700 1 Processor -- start 2024-01-15T14:35:34.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f348de65700 1 -- start start 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f348de65700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3488161690 0x7f34880d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f348de65700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f348809b110 0x7f34880d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f348de65700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34880d0a70 con 0x7f3488161a70 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f348de65700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34880d1c10 con 0x7f3488161690 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f348de65700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34880b2ab0 con 0x7f348809b110 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.400+0000 7f34877fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3488161690 0x7f34880d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3488161690 0x7f34880d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57788/0 (socket says 172.21.15.77:57788) 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 -- 172.21.15.77:0/2123665441 learned_addr learned my addr 172.21.15.77:0/2123665441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f3487fff700 1 --2- 172.21.15.77:0/2123665441 >> v2:172.21.15.159:3300/0 conn(0x7f348809b110 0x7f34880d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 -- 172.21.15.77:0/2123665441 >> v2:172.21.15.159:3300/0 conn(0x7f348809b110 msgr2=0x7f34880d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 --2- 172.21.15.77:0/2123665441 >> v2:172.21.15.159:3300/0 conn(0x7f348809b110 0x7f34880d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 -- 172.21.15.77:0/2123665441 >> v1:172.21.15.159:6789/0 conn(0x7f3488161a70 legacy=0x7f348809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 -- 172.21.15.77:0/2123665441 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34880b2d90 con 0x7f3488161690 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f34877fe700 1 --2- 172.21.15.77:0/2123665441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3488161690 0x7f34880d0f20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f34880ce460 tx=0x7f3478009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c799c51ee2553531 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f3486ffd700 1 -- 172.21.15.77:0/2123665441 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3478011830 con 0x7f3488161690 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f3486ffd700 1 -- 172.21.15.77:0/2123665441 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3478011990 con 0x7f3488161690 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.401+0000 7f3486ffd700 1 -- 172.21.15.77:0/2123665441 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3478011d20 con 0x7f3488161690 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 -- 172.21.15.77:0/2123665441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3488161690 msgr2=0x7f34880d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 --2- 172.21.15.77:0/2123665441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3488161690 0x7f34880d0f20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f34880ce460 tx=0x7f3478009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 -- 172.21.15.77:0/2123665441 shutdown_connections 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 --2- 172.21.15.77:0/2123665441 >> v2:172.21.15.159:3300/0 conn(0x7f348809b110 0x7f34880d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 --2- 172.21.15.77:0/2123665441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3488161690 0x7f34880d0f20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 -- 172.21.15.77:0/2123665441 >> 172.21.15.77:0/2123665441 conn(0x7f34880d1e50 msgr2=0x7f34880d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 -- 172.21.15.77:0/2123665441 shutdown_connections 2024-01-15T14:35:34.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.402+0000 7f348de65700 1 -- 172.21.15.77:0/2123665441 wait complete. 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.403+0000 7f348de65700 1 Processor -- start 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.403+0000 7f348de65700 1 -- start start 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 0x7f348815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3488161690 0x7f34880b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3488161a70 0x7f34880b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34880c5cf0 con 0x7f348809b110 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34880c62b0 con 0x7f3488161690 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34880c6590 con 0x7f3488161a70 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 0x7f348815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 0x7f348815b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57790/0 (socket says 172.21.15.77:57790) 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 -- 172.21.15.77:0/3716273554 learned_addr learned my addr 172.21.15.77:0/3716273554 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f3487fff700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3488161a70 0x7f34880b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f347ffff700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3488161690 0x7f34880b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 -- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3488161a70 msgr2=0x7f34880b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3488161a70 0x7f34880b5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 -- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3488161690 msgr2=0x7f34880b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3488161690 0x7f34880b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 -- 172.21.15.77:0/3716273554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478009550 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f347ffff700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3488161690 0x7f34880b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f34877fe700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 0x7f348815b710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3478009c10 tx=0x7f3478004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34780036a0 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347801e070 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34880b4c00 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3478003a60 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.404+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34880c5980 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.405+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3488069020 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.406+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3478003bc0 con 0x7f348809b110 2024-01-15T14:35:34.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.406+0000 7f3484ff9700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f346806b350 0x7f346806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.406+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f347808aaa0 con 0x7f348809b110 2024-01-15T14:35:34.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.406+0000 7f347ffff700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f346806b350 0x7f346806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.407+0000 7f347ffff700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f346806b350 0x7f346806d810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f34880c6260 tx=0x7f3470009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.411+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34780559a0 con 0x7f348809b110 2024-01-15T14:35:34.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.599+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34880b5db0 con 0x7f348809b110 2024-01-15T14:35:34.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.599+0000 7f3484ff9700 1 -- 172.21.15.77:0/3716273554 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3478058fc0 con 0x7f348809b110 2024-01-15T14:35:34.600 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f346806b350 msgr2=0x7f346806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f346806b350 0x7f346806d810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f34880c6260 tx=0x7f3470009040 comp rx=0 tx=0).stop 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 msgr2=0x7f348815b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 0x7f348815b710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3478009c10 tx=0x7f3478004750 comp rx=0 tx=0).stop 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 shutdown_connections 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f346806b350 0x7f346806d810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3488161a70 0x7f34880b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348809b110 0x7f348815b710 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 --2- 172.21.15.77:0/3716273554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3488161690 0x7f34880b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 >> 172.21.15.77:0/3716273554 conn(0x7f34880d1e50 msgr2=0x7f34880d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 shutdown_connections 2024-01-15T14:35:34.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.601+0000 7f348de65700 1 -- 172.21.15.77:0/3716273554 wait complete. 2024-01-15T14:35:34.614 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.737+0000 7fd37ad2e700 1 Processor -- start 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.737+0000 7fd37ad2e700 1 -- start start 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd37ad2e700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd37415f8d0 0x7fd3740b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd37ad2e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd37ad2e700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740b3860 con 0x7fd3740cffc0 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd37ad2e700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3740b2520 con 0x7fd37415f8d0 2024-01-15T14:35:34.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd37ad2e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740b2800 con 0x7fd3740d03a0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd3792cb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd373fff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd37415f8d0 0x7fd3740b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd373fff700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd37415f8d0 0x7fd3740b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:43988/0 (socket says 172.21.15.77:43988) 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd3792cb700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:59886/0 (socket says 172.21.15.77:59886) 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd373fff700 1 -- 172.21.15.77:0/1019503696 learned_addr learned my addr 172.21.15.77:0/1019503696 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.738+0000 7fd378aca700 1 --1- 172.21.15.77:0/1019503696 >> v1:172.21.15.159:6789/0 conn(0x7fd3740cffc0 0x7fd37415f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:54210/0 (socket says 172.21.15.77:54210) 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd373fff700 1 -- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 msgr2=0x7fd3740ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd373fff700 1 --2- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd373fff700 1 -- 172.21.15.77:0/1019503696 >> v1:172.21.15.159:6789/0 conn(0x7fd3740cffc0 legacy=0x7fd37415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd373fff700 1 -- 172.21.15.77:0/1019503696 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740b2ae0 con 0x7fd37415f8d0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd373fff700 1 --2- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd37415f8d0 0x7fd3740b3320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd3600098f0 tx=0x7fd360009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4a19fe4cb8cdb9c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd3737fe700 1 -- 172.21.15.77:0/1019503696 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360011830 con 0x7fd37415f8d0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd3737fe700 1 -- 172.21.15.77:0/1019503696 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360011990 con 0x7fd37415f8d0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.739+0000 7fd3737fe700 1 -- 172.21.15.77:0/1019503696 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360011d20 con 0x7fd37415f8d0 2024-01-15T14:35:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 -- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd37415f8d0 msgr2=0x7fd3740b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd37415f8d0 0x7fd3740b3320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd3600098f0 tx=0x7fd360009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 -- 172.21.15.77:0/1019503696 shutdown_connections 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/1019503696 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd37415f8d0 0x7fd3740b3320 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 -- 172.21.15.77:0/1019503696 >> 172.21.15.77:0/1019503696 conn(0x7fd3740d1e50 msgr2=0x7fd3740d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 -- 172.21.15.77:0/1019503696 shutdown_connections 2024-01-15T14:35:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.740+0000 7fd37ad2e700 1 -- 172.21.15.77:0/1019503696 wait complete. 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.741+0000 7fd37ad2e700 1 Processor -- start 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.741+0000 7fd37ad2e700 1 -- start start 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd37ad2e700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740cffc0 0x7fd37415c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd37ad2e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd37ad2e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 0x7fd3740b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd37ad2e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740c5fd0 con 0x7fd37415f8d0 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd37ad2e700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37406bc70 con 0x7fd3740cffc0 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd37ad2e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37406bf50 con 0x7fd3740d03a0 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 0x7fd3740b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 0x7fd3740b6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57806/0 (socket says 172.21.15.77:57806) 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 -- 172.21.15.77:0/2970933396 learned_addr learned my addr 172.21.15.77:0/2970933396 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd373fff700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd378aca700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740cffc0 0x7fd37415c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 -- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 msgr2=0x7fd3740c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 -- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740cffc0 msgr2=0x7fd37415c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740cffc0 0x7fd37415c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 -- 172.21.15.77:0/2970933396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360009550 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd378aca700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740cffc0 0x7fd37415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.742+0000 7fd3792cb700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 0x7fd3740b6cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00efc0 tx=0x7fd36c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.743+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c009ca0 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.743+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c00f040 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.743+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3740c5980 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.743+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c014480 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.743+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37415cbd0 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.743+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3740c6140 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.744+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd36c0145e0 con 0x7fd37415f8d0 2024-01-15T14:35:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.744+0000 7fd3717fa700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35806b350 0x7fd35806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.744+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd36c08a830 con 0x7fd37415f8d0 2024-01-15T14:35:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.744+0000 7fd378aca700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35806b350 0x7fd35806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.745+0000 7fd378aca700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35806b350 0x7fd35806d810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd368000f60 tx=0x7fd368009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.749+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd36c0556b0 con 0x7fd37415f8d0 2024-01-15T14:35:34.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.940+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd374069020 con 0x7fd37415f8d0 2024-01-15T14:35:34.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.941+0000 7fd3717fa700 1 -- 172.21.15.77:0/2970933396 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd36c0180a0 con 0x7fd37415f8d0 2024-01-15T14:35:34.942 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35806b350 msgr2=0x7fd35806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35806b350 0x7fd35806d810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd368000f60 tx=0x7fd368009040 comp rx=0 tx=0).stop 2024-01-15T14:35:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 msgr2=0x7fd3740b6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 0x7fd3740b6cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00efc0 tx=0x7fd36c00c3a0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 shutdown_connections 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35806b350 0x7fd35806d810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd37415f8d0 0x7fd3740b6cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740cffc0 0x7fd37415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 --2- 172.21.15.77:0/2970933396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740d03a0 0x7fd3740c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 >> 172.21.15.77:0/2970933396 conn(0x7fd3740d1e50 msgr2=0x7fd37415d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 shutdown_connections 2024-01-15T14:35:34.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:34.943+0000 7fd37ad2e700 1 -- 172.21.15.77:0/2970933396 wait complete. 2024-01-15T14:35:34.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-15T14:35:35.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.076+0000 7f9b9e31d700 1 Processor -- start 2024-01-15T14:35:35.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.076+0000 7f9b9e31d700 1 -- start start 2024-01-15T14:35:35.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b9e31d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9b9815af10 0x7f9b9815e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:35.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b9e31d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b981414c0 con 0x7f9b9815f630 2024-01-15T14:35:35.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b9e31d700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b980ce9f0 con 0x7f9b980ce140 2024-01-15T14:35:35.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b9e31d700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9815c300 con 0x7f9b9815af10 2024-01-15T14:35:35.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b9c8ba700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9b9815f630 0x7f9b9806b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43568/0 (socket says 172.21.15.77:43568) 2024-01-15T14:35:35.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b9c8ba700 1 -- 172.21.15.77:0/3078443748 learned_addr learned my addr 172.21.15.77:0/3078443748 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b97fff700 1 --2- 172.21.15.77:0/3078443748 >> v2:172.21.15.159:3300/0 conn(0x7f9b9815af10 0x7f9b9815e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b96ffd700 1 -- 172.21.15.77:0/3078443748 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628187529 0 0) 0x7f9b981414c0 con 0x7f9b9815f630 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b96ffd700 1 -- 172.21.15.77:0/3078443748 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b800035b0 con 0x7f9b9815f630 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b96ffd700 1 -- 172.21.15.77:0/3078443748 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4128644359 0 0) 0x7f9b800035b0 con 0x7f9b9815f630 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b97fff700 1 -- 172.21.15.77:0/3078443748 >> v1:172.21.15.159:6789/0 conn(0x7f9b980ce140 legacy=0x7f9b9815ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b97fff700 1 -- 172.21.15.77:0/3078443748 >> v1:172.21.15.77:6789/0 conn(0x7f9b9815f630 legacy=0x7f9b9806b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.077+0000 7f9b97fff700 1 -- 172.21.15.77:0/3078443748 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b980cecd0 con 0x7f9b9815af10 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b97fff700 1 --2- 172.21.15.77:0/3078443748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b9815af10 0x7f9b9815e9e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009840 tx=0x7f9b88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fad323cf0c5f1a41 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b96ffd700 1 -- 172.21.15.77:0/3078443748 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900019f0 con 0x7f9b9815af10 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b96ffd700 1 -- 172.21.15.77:0/3078443748 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b88004030 con 0x7f9b9815af10 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b96ffd700 1 -- 172.21.15.77:0/3078443748 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880043c0 con 0x7f9b9815af10 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 -- 172.21.15.77:0/3078443748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b9815af10 msgr2=0x7f9b9815e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/3078443748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b9815af10 0x7f9b9815e9e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009840 tx=0x7f9b88009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 -- 172.21.15.77:0/3078443748 shutdown_connections 2024-01-15T14:35:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/3078443748 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b9815af10 0x7f9b9815e9e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 -- 172.21.15.77:0/3078443748 >> 172.21.15.77:0/3078443748 conn(0x7f9b980cd570 msgr2=0x7f9b980cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 -- 172.21.15.77:0/3078443748 shutdown_connections 2024-01-15T14:35:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.078+0000 7f9b9e31d700 1 -- 172.21.15.77:0/3078443748 wait complete. 2024-01-15T14:35:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.080+0000 7f9b9e31d700 1 Processor -- start 2024-01-15T14:35:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.080+0000 7f9b9e31d700 1 -- start start 2024-01-15T14:35:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9e31d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b980ce140 0x7f9b980afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9e31d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 0x7f9b981596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9e31d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b980ad870 0x7f9b980b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9e31d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980ae980 con 0x7f9b9815af10 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9e31d700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980b5600 con 0x7f9b980ce140 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9e31d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980add00 con 0x7f9b980ad870 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 0x7f9b981596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 0x7f9b981596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57820/0 (socket says 172.21.15.77:57820) 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 -- 172.21.15.77:0/1718007768 learned_addr learned my addr 172.21.15.77:0/1718007768 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b97fff700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b980ce140 0x7f9b980afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b9c8ba700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b980ad870 0x7f9b980b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 -- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b980ad870 msgr2=0x7f9b980b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b980ad870 0x7f9b980b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 -- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b980ce140 msgr2=0x7f9b980afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b980ce140 0x7f9b980afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 -- 172.21.15.77:0/1718007768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b880094a0 con 0x7f9b9815af10 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b97fff700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b980ce140 0x7f9b980afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.081+0000 7f9b977fe700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 0x7f9b981596c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00c4b0 tx=0x7f9b8c00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.082+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0052d0 con 0x7f9b9815af10 2024-01-15T14:35:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.082+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c005430 con 0x7f9b9815af10 2024-01-15T14:35:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.082+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b980adfe0 con 0x7f9b9815af10 2024-01-15T14:35:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.082+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0038c0 con 0x7f9b9815af10 2024-01-15T14:35:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.082+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b980ae5a0 con 0x7f9b9815af10 2024-01-15T14:35:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.083+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b980aeac0 con 0x7f9b9815af10 2024-01-15T14:35:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.083+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9b8c003a20 con 0x7f9b9815af10 2024-01-15T14:35:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.083+0000 7f9b94ff9700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b8406b350 0x7f9b8406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.084+0000 7f9b97fff700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b8406b350 0x7f9b8406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.084+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b8c088fc0 con 0x7f9b9815af10 2024-01-15T14:35:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.084+0000 7f9b97fff700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b8406b350 0x7f9b8406d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9b880090b0 tx=0x7f9b88008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.088+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b8c0bbe90 con 0x7f9b9815af10 2024-01-15T14:35:35.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:35.360+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f9b98068670 con 0x7f9b9815af10 2024-01-15T14:35:36.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.206+0000 7f9b94ff9700 1 -- 172.21.15.77:0/1718007768 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v54) v1 ==== 155+0+0 (secure 0 0 0) 0x7f9b8c0170e0 con 0x7f9b9815af10 2024-01-15T14:35:36.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b8406b350 msgr2=0x7f9b8406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b8406b350 0x7f9b8406d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9b880090b0 tx=0x7f9b88008030 comp rx=0 tx=0).stop 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 msgr2=0x7f9b981596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 0x7f9b981596c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00c4b0 tx=0x7f9b8c00c870 comp rx=0 tx=0).stop 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 shutdown_connections 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9b8406b350 0x7f9b8406d810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9b980ad870 0x7f9b980b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9b980ce140 0x7f9b980afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 --2- 172.21.15.77:0/1718007768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9815af10 0x7f9b981596c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 >> 172.21.15.77:0/1718007768 conn(0x7f9b980cd570 msgr2=0x7f9b980b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 shutdown_connections 2024-01-15T14:35:36.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.209+0000 7f9b9e31d700 1 -- 172.21.15.77:0/1718007768 wait complete. 2024-01-15T14:35:36.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T14:35:36.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.350+0000 7f18cadd2700 1 Processor -- start 2024-01-15T14:35:36.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.350+0000 7f18cadd2700 1 -- start start 2024-01-15T14:35:36.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18cadd2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c415e0c0 0x7f18c40afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18cadd2700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f18c40b3bf0 0x7f18c4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18cadd2700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18c40744d0 0x7f18c4161c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18cadd2700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c4074960 con 0x7f18c40744d0 2024-01-15T14:35:36.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18cadd2700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c415e4a0 con 0x7f18c40b3bf0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18cadd2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c406aff0 con 0x7f18c415e0c0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18c40744d0 0x7f18c4161c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18c40744d0 0x7f18c4161c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57822/0 (socket says 172.21.15.77:57822) 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 -- 172.21.15.77:0/1490329550 learned_addr learned my addr 172.21.15.77:0/1490329550 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 -- 172.21.15.77:0/1490329550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c415e0c0 msgr2=0x7f18c40afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 --2- 172.21.15.77:0/1490329550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c415e0c0 0x7f18c40afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 -- 172.21.15.77:0/1490329550 >> v2:172.21.15.159:3300/0 conn(0x7f18c40b3bf0 msgr2=0x7f18c4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 --2- 172.21.15.77:0/1490329550 >> v2:172.21.15.159:3300/0 conn(0x7f18c40b3bf0 0x7f18c4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 -- 172.21.15.77:0/1490329550 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c406b2d0 con 0x7f18c40744d0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c936f700 1 --2- 172.21.15.77:0/1490329550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c4161c10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f18c409f090 tx=0x7f18bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05fb71f56650e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c37fe700 1 -- 172.21.15.77:0/1490329550 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18bc00e070 con 0x7f18c40744d0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c37fe700 1 -- 172.21.15.77:0/1490329550 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18bc00b9c0 con 0x7f18c40744d0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.351+0000 7f18c37fe700 1 -- 172.21.15.77:0/1490329550 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18bc00bd70 con 0x7f18c40744d0 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 -- 172.21.15.77:0/1490329550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 msgr2=0x7f18c4161c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 --2- 172.21.15.77:0/1490329550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c4161c10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f18c409f090 tx=0x7f18bc00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 -- 172.21.15.77:0/1490329550 shutdown_connections 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 --2- 172.21.15.77:0/1490329550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c4161c10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 --2- 172.21.15.77:0/1490329550 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c415e0c0 0x7f18c40afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 --2- 172.21.15.77:0/1490329550 >> v2:172.21.15.159:3300/0 conn(0x7f18c40b3bf0 0x7f18c4073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 -- 172.21.15.77:0/1490329550 >> 172.21.15.77:0/1490329550 conn(0x7f18c40d0c00 msgr2=0x7f18c40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 -- 172.21.15.77:0/1490329550 shutdown_connections 2024-01-15T14:35:36.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.352+0000 7f18cadd2700 1 -- 172.21.15.77:0/1490329550 wait complete. 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.353+0000 7f18cadd2700 1 Processor -- start 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.353+0000 7f18cadd2700 1 -- start start 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18cadd2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c40ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18cadd2700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c40b3bf0 0x7f18c41f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18cadd2700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c415e0c0 0x7f18c4156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18cadd2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4155c90 con 0x7f18c40744d0 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18cadd2700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4155f30 con 0x7f18c415e0c0 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18cadd2700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4156210 con 0x7f18c40b3bf0 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c8b6e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c40ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c8b6e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c40ce320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57840/0 (socket says 172.21.15.77:57840) 2024-01-15T14:35:36.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c8b6e700 1 -- 172.21.15.77:0/581746777 learned_addr learned my addr 172.21.15.77:0/581746777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c936f700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c415e0c0 0x7f18c4156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c3fff700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c40b3bf0 0x7f18c41f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c8b6e700 1 -- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c40b3bf0 msgr2=0x7f18c41f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c8b6e700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c40b3bf0 0x7f18c41f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.354+0000 7f18c8b6e700 1 -- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c415e0c0 msgr2=0x7f18c4156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c8b6e700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c415e0c0 0x7f18c4156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c8b6e700 1 -- 172.21.15.77:0/581746777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc009b30 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c3fff700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c40b3bf0 0x7f18c41f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c936f700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c415e0c0 0x7f18c4156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c8b6e700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c40ce320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f18b800b530 tx=0x7f18b800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8011830 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b80119b0 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c41564f0 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8011d60 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.355+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c40cd350 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.356+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4156a10 con 0x7f18c40744d0 2024-01-15T14:35:36.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.356+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f18b801a400 con 0x7f18c40744d0 2024-01-15T14:35:36.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.357+0000 7f18c17fa700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f18a806b350 0x7f18a806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.357+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18b8089c20 con 0x7f18c40744d0 2024-01-15T14:35:36.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.357+0000 7f18c3fff700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f18a806b350 0x7f18a806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.357+0000 7f18c3fff700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f18a806b350 0x7f18a806d810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f18c40b04e0 tx=0x7f18b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.361+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18c4156a10 con 0x7f18c40744d0 2024-01-15T14:35:36.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.552+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f18c4156a10 con 0x7f18c40744d0 2024-01-15T14:35:36.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.555+0000 7f18c17fa700 1 -- 172.21.15.77:0/581746777 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f18c4156a10 con 0x7f18c40744d0 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.557+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f18a806b350 msgr2=0x7f18a806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.557+0000 7f18cadd2700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f18a806b350 0x7f18a806d810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f18c40b04e0 tx=0x7f18b0008040 comp rx=0 tx=0).stop 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.557+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 msgr2=0x7f18c40ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.557+0000 7f18cadd2700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c40ce320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f18b800b530 tx=0x7f18b800b8f0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 shutdown_connections 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f18a806b350 0x7f18a806d810 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f18c415e0c0 0x7f18c4156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18c40744d0 0x7f18c40ce320 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 --2- 172.21.15.77:0/581746777 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f18c40b3bf0 0x7f18c41f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 >> 172.21.15.77:0/581746777 conn(0x7f18c40d0c00 msgr2=0x7f18c40add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 shutdown_connections 2024-01-15T14:35:36.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.558+0000 7f18cadd2700 1 -- 172.21.15.77:0/581746777 wait complete. 2024-01-15T14:35:36.571 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T14:35:36.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T14:35:36.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T14:35:36.666 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T14:35:36.666 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:36.666 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:36.683 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:36.683 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:36.741 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:36.741 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:36.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:36.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.959+0000 7f5fbe82b700 1 Processor -- start 2024-01-15T14:35:36.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.959+0000 7f5fbe82b700 1 -- start start 2024-01-15T14:35:36.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fbe82b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fb80d2160 0x7f5fb80b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fbe82b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb80d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fbe82b700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb80d0770 con 0x7f5fb815bed0 2024-01-15T14:35:36.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fbe82b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb815c2b0 con 0x7f5fb80d2160 2024-01-15T14:35:36.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fbe82b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb80cd100 con 0x7f5fb81603c0 2024-01-15T14:35:36.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fb80d2160 0x7f5fb80b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb7fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5fb815bed0 0x7f5fb80d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43574/0 (socket says 172.21.15.77:43574) 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb7fff700 1 -- 172.21.15.77:0/1110553342 learned_addr learned my addr 172.21.15.77:0/1110553342 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb77fe700 1 -- 172.21.15.77:0/1110553342 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 msgr2=0x7f5fb80d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/1110553342 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb80d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb77fe700 1 -- 172.21.15.77:0/1110553342 >> v1:172.21.15.77:6789/0 conn(0x7f5fb815bed0 legacy=0x7f5fb80d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb77fe700 1 -- 172.21.15.77:0/1110553342 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb8160870 con 0x7f5fb80d2160 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb6ffd700 1 -- 172.21.15.77:0/1110553342 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341099372 0 0) 0x7f5fb80d0770 con 0x7f5fb815bed0 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.960+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/1110553342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b19f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009840 tx=0x7f5fa0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee55130a6d59a724 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fb6ffd700 1 -- 172.21.15.77:0/1110553342 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0011830 con 0x7f5fb80d2160 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fb6ffd700 1 -- 172.21.15.77:0/1110553342 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa0011990 con 0x7f5fb80d2160 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fb6ffd700 1 -- 172.21.15.77:0/1110553342 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0011d20 con 0x7f5fb80d2160 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 -- 172.21.15.77:0/1110553342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb80d2160 msgr2=0x7f5fb80b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/1110553342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b19f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009840 tx=0x7f5fa0009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 -- 172.21.15.77:0/1110553342 shutdown_connections 2024-01-15T14:35:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/1110553342 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/1110553342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b19f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 -- 172.21.15.77:0/1110553342 >> 172.21.15.77:0/1110553342 conn(0x7f5fb8073130 msgr2=0x7f5fb809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:36.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 -- 172.21.15.77:0/1110553342 shutdown_connections 2024-01-15T14:35:36.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.961+0000 7f5fbe82b700 1 -- 172.21.15.77:0/1110553342 wait complete. 2024-01-15T14:35:36.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.962+0000 7f5fbe82b700 1 Processor -- start 2024-01-15T14:35:36.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.962+0000 7f5fbe82b700 1 -- start start 2024-01-15T14:35:36.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbe82b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbe82b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 0x7f5fb80b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbe82b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb815cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbe82b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb815e400 con 0x7f5fb815bed0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbe82b700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb815d7f0 con 0x7f5fb80d2160 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbe82b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb815dad0 con 0x7f5fb81603c0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fb77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 0x7f5fb80b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fb77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 0x7f5fb80b7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57848/0 (socket says 172.21.15.77:57848) 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fb77fe700 1 -- 172.21.15.77:0/564893425 learned_addr learned my addr 172.21.15.77:0/564893425 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fbcdc8700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb815cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.963+0000 7f5fb7fff700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb77fe700 1 -- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 msgr2=0x7f5fb815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb815cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb77fe700 1 -- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fb80d2160 msgr2=0x7f5fb80b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb77fe700 1 -- 172.21.15.77:0/564893425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00094a0 con 0x7f5fb815bed0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 0x7f5fb80b7630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00090b0 tx=0x7f5fa0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00045f0 con 0x7f5fb815bed0 2024-01-15T14:35:36.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa001f070 con 0x7f5fb815bed0 2024-01-15T14:35:36.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb815ddb0 con 0x7f5fb815bed0 2024-01-15T14:35:36.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0003b00 con 0x7f5fb815bed0 2024-01-15T14:35:36.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.964+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb81592d0 con 0x7f5fb815bed0 2024-01-15T14:35:36.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.965+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb8067f60 con 0x7f5fb815bed0 2024-01-15T14:35:36.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.965+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5fa0003c60 con 0x7f5fb815bed0 2024-01-15T14:35:36.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.966+0000 7f5fb4ff9700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f9806b350 0x7f5f9806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:36.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.966+0000 7f5fb7fff700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f9806b350 0x7f5f9806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:36.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.966+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fa0021070 con 0x7f5fb815bed0 2024-01-15T14:35:36.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.966+0000 7f5fb7fff700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f9806b350 0x7f5f9806d810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8000f60 tx=0x7f5fa8007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:36.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:36.970+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fa0056540 con 0x7f5fb815bed0 2024-01-15T14:35:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.163+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fb815b6f0 con 0x7f5fb815bed0 2024-01-15T14:35:37.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.164+0000 7f5fb4ff9700 1 -- 172.21.15.77:0/564893425 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f5fa0059b60 con 0x7f5fb815bed0 2024-01-15T14:35:37.165 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:37.165 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:37.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f9806b350 msgr2=0x7f5f9806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f9806b350 0x7f5f9806d810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8000f60 tx=0x7f5fa8007be0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 msgr2=0x7f5fb80b7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 0x7f5fb80b7630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00090b0 tx=0x7f5fa0004750 comp rx=0 tx=0).stop 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 shutdown_connections 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f9806b350 0x7f5f9806d810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5fb81603c0 0x7f5fb815cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5fb80d2160 0x7f5fb80b70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 --2- 172.21.15.77:0/564893425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb815bed0 0x7f5fb80b7630 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 >> 172.21.15.77:0/564893425 conn(0x7f5fb8073130 msgr2=0x7f5fb80d1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 shutdown_connections 2024-01-15T14:35:37.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.166+0000 7f5fbe82b700 1 -- 172.21.15.77:0/564893425 wait complete. 2024-01-15T14:35:37.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:37.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.305+0000 7f3f13f72700 1 Processor -- start 2024-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 -- start start 2024-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f0c0d4590 0x7f3f0c0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3f0c0b7540 0x7f3f0c0d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d23e0 con 0x7f3f0c0d4590 2024-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c0d5010 con 0x7f3f0c0b7540 2024-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.306+0000 7f3f13f72700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c15e4f0 con 0x7f3f0c15e110 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f0c0d4590 0x7f3f0c0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f0c0d4590 0x7f3f0c0b7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57860/0 (socket says 172.21.15.77:57860) 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 -- 172.21.15.77:0/1817364751 learned_addr learned my addr 172.21.15.77:0/1817364751 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 -- 172.21.15.77:0/1817364751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 msgr2=0x7f3f0c0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 --2- 172.21.15.77:0/1817364751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 -- 172.21.15.77:0/1817364751 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c0b7540 msgr2=0x7f3f0c0d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1250f700 1 --2- 172.21.15.77:0/1817364751 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c0b7540 0x7f3f0c0d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 --2- 172.21.15.77:0/1817364751 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c0b7540 0x7f3f0c0d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 -- 172.21.15.77:0/1817364751 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0d4d30 con 0x7f3f0c0d4590 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1250f700 1 --2- 172.21.15.77:0/1817364751 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c0b7540 0x7f3f0c0d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f1150d700 1 --2- 172.21.15.77:0/1817364751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b7000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c161b70 tx=0x7f3f08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b738ef1298ce5bfa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f10d0c700 1 -- 172.21.15.77:0/1817364751 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800b870 con 0x7f3f0c0d4590 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f10d0c700 1 -- 172.21.15.77:0/1817364751 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0800b9d0 con 0x7f3f0c0d4590 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.307+0000 7f3f10d0c700 1 -- 172.21.15.77:0/1817364751 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800bd60 con 0x7f3f0c0d4590 2024-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 -- 172.21.15.77:0/1817364751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 msgr2=0x7f3f0c0b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1817364751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b7000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c161b70 tx=0x7f3f08009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 -- 172.21.15.77:0/1817364751 shutdown_connections 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1817364751 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c0b7540 0x7f3f0c0d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1817364751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1817364751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b7000 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 -- 172.21.15.77:0/1817364751 >> 172.21.15.77:0/1817364751 conn(0x7f3f0c071a30 msgr2=0x7f3f0c071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 -- 172.21.15.77:0/1817364751 shutdown_connections 2024-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.308+0000 7f3f13f72700 1 -- 172.21.15.77:0/1817364751 wait complete. 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.309+0000 7f3f13f72700 1 Processor -- start 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.309+0000 7f3f13f72700 1 -- start start 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f13f72700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0b7540 0x7f3f0c150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f13f72700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f13f72700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f13f72700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c1512c0 con 0x7f3f0c0d4590 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f13f72700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0b4590 con 0x7f3f0c0b7540 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f13f72700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0b4870 con 0x7f3f0c15e110 2024-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f1150d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f1150d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57874/0 (socket says 172.21.15.77:57874) 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.310+0000 7f3f1150d700 1 -- 172.21.15.77:0/1388942666 learned_addr learned my addr 172.21.15.77:0/1388942666 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1250f700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f11d0e700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0b7540 0x7f3f0c150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1150d700 1 -- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 msgr2=0x7f3f0c0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1150d700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1150d700 1 -- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0b7540 msgr2=0x7f3f0c150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1150d700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0b7540 0x7f3f0c150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1150d700 1 -- 172.21.15.77:0/1388942666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f08009550 con 0x7f3f0c0d4590 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f1150d700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b3b10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0d1c30 tx=0x7f3f08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08004310 con 0x7f3f0c0d4590 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0801f070 con 0x7f3f0c0d4590 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c0b4b50 con 0x7f3f0c0d4590 2024-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f080036a0 con 0x7f3f0c0d4590 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.311+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c0b5110 con 0x7f3f0c0d4590 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.312+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c069020 con 0x7f3f0c0d4590 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.312+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3f08021030 con 0x7f3f0c0d4590 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.313+0000 7f3efe7fc700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.313+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f0808b170 con 0x7f3f0c0d4590 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.313+0000 7f3f11d0e700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.313+0000 7f3f11d0e700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3f00006fd0 tx=0x7f3f00008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.317+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f080560f0 con 0x7f3f0c0d4590 2024-01-15T14:35:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.506+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f0c0cf420 con 0x7f3f0c0d4590 2024-01-15T14:35:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.507+0000 7f3efe7fc700 1 -- 172.21.15.77:0/1388942666 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f08059710 con 0x7f3f0c0d4590 2024-01-15T14:35:37.508 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 msgr2=0x7f3ef006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3f00006fd0 tx=0x7f3f00008040 comp rx=0 tx=0).stop 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 msgr2=0x7f3f0c0b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b3b10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0d1c30 tx=0x7f3f08004750 comp rx=0 tx=0).stop 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 shutdown_connections 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c15e110 0x7f3f0c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0b7540 0x7f3f0c150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.508+0000 7f3f13f72700 1 --2- 172.21.15.77:0/1388942666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c0d4590 0x7f3f0c0b3b10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.509+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 >> 172.21.15.77:0/1388942666 conn(0x7f3f0c071a30 msgr2=0x7f3f0c0d3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.509+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 shutdown_connections 2024-01-15T14:35:37.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.509+0000 7f3f13f72700 1 -- 172.21.15.77:0/1388942666 wait complete. 2024-01-15T14:35:37.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.644+0000 7f01c432d700 1 Processor -- start 2024-01-15T14:35:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.644+0000 7f01c432d700 1 -- start start 2024-01-15T14:35:37.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c432d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01bc15e110 0x7f01bc06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c432d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f01bc06e420 0x7f01bc06e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c432d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc0affe0 con 0x7f01bc0afb30 2024-01-15T14:35:37.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c432d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc06d980 con 0x7f01bc15e110 2024-01-15T14:35:37.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c432d700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc0b0260 con 0x7f01bc06e420 2024-01-15T14:35:37.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c20c9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01bc15e110 0x7f01bc06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c20c9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01bc15e110 0x7f01bc06f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57876/0 (socket says 172.21.15.77:57876) 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c20c9700 1 -- 172.21.15.77:0/1754298397 learned_addr learned my addr 172.21.15.77:0/1754298397 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.645+0000 7f01c18c8700 1 --1- 172.21.15.77:0/1754298397 >> v1:172.21.15.77:6789/0 conn(0x7f01bc0afb30 0x7f01bc06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43576/0 (socket says 172.21.15.77:43576) 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c28ca700 1 --2- 172.21.15.77:0/1754298397 >> v2:172.21.15.159:3300/0 conn(0x7f01bc06e420 0x7f01bc06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c20c9700 1 -- 172.21.15.77:0/1754298397 >> v2:172.21.15.159:3300/0 conn(0x7f01bc06e420 msgr2=0x7f01bc06e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c20c9700 1 --2- 172.21.15.77:0/1754298397 >> v2:172.21.15.159:3300/0 conn(0x7f01bc06e420 0x7f01bc06e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c20c9700 1 -- 172.21.15.77:0/1754298397 >> v1:172.21.15.77:6789/0 conn(0x7f01bc0afb30 legacy=0x7f01bc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c20c9700 1 -- 172.21.15.77:0/1754298397 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0b0540 con 0x7f01bc15e110 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c28ca700 1 --2- 172.21.15.77:0/1754298397 >> v2:172.21.15.159:3300/0 conn(0x7f01bc06e420 0x7f01bc06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c10c7700 1 -- 172.21.15.77:0/1754298397 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447105929 0 0) 0x7f01bc0affe0 con 0x7f01bc0afb30 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c20c9700 1 --2- 172.21.15.77:0/1754298397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc06f680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f01bc071b90 tx=0x7f01ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c51dd16b3ce235b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c10c7700 1 -- 172.21.15.77:0/1754298397 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac00b870 con 0x7f01bc15e110 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c10c7700 1 -- 172.21.15.77:0/1754298397 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac00b9d0 con 0x7f01bc15e110 2024-01-15T14:35:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c10c7700 1 -- 172.21.15.77:0/1754298397 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac00bd60 con 0x7f01bc15e110 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c432d700 1 -- 172.21.15.77:0/1754298397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 msgr2=0x7f01bc06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.646+0000 7f01c432d700 1 --2- 172.21.15.77:0/1754298397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc06f680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f01bc071b90 tx=0x7f01ac009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.647+0000 7f01c432d700 1 -- 172.21.15.77:0/1754298397 shutdown_connections 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.647+0000 7f01c432d700 1 --2- 172.21.15.77:0/1754298397 >> v2:172.21.15.159:3300/0 conn(0x7f01bc06e420 0x7f01bc06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.647+0000 7f01c432d700 1 --2- 172.21.15.77:0/1754298397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc06f680 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.647+0000 7f01c432d700 1 -- 172.21.15.77:0/1754298397 >> 172.21.15.77:0/1754298397 conn(0x7f01bc0b3250 msgr2=0x7f01bc0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.647+0000 7f01c432d700 1 -- 172.21.15.77:0/1754298397 shutdown_connections 2024-01-15T14:35:37.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.647+0000 7f01c432d700 1 -- 172.21.15.77:0/1754298397 wait complete. 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.648+0000 7f01c432d700 1 Processor -- start 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.648+0000 7f01c432d700 1 -- start start 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.648+0000 7f01c432d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01bc06e420 0x7f01bc1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.648+0000 7f01c432d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01bc0afb30 0x7f01bc157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c432d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc15a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c432d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc1596c0 con 0x7f01bc15e110 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c432d700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc158ab0 con 0x7f01bc0afb30 2024-01-15T14:35:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c432d700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc158d90 con 0x7f01bc06e420 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc15a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc15a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57882/0 (socket says 172.21.15.77:57882) 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 -- 172.21.15.77:0/1531990584 learned_addr learned my addr 172.21.15.77:0/1531990584 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c18c8700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01bc0afb30 0x7f01bc157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c20c9700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01bc06e420 0x7f01bc1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 -- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01bc06e420 msgr2=0x7f01bc1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01bc06e420 0x7f01bc1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 -- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01bc0afb30 msgr2=0x7f01bc157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01bc0afb30 0x7f01bc157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 -- 172.21.15.77:0/1531990584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac009550 con 0x7f01bc15e110 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c20c9700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01bc06e420 0x7f01bc1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c28ca700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc15a760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f01b400cef0 tx=0x7f01b400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400ae50 con 0x7f01bc15e110 2024-01-15T14:35:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b4015440 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc06b880 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40157f0 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.649+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc159350 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.650+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc1615a0 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.651+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f01b400d430 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.651+0000 7f01b2ffd700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f01a006b380 0x7f01a006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.651+0000 7f01c20c9700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f01a006b380 0x7f01a006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.651+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01b408a170 con 0x7f01bc15e110 2024-01-15T14:35:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.652+0000 7f01c20c9700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f01a006b380 0x7f01a006d840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f01ac000c00 tx=0x7f01ac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.656+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01b4054fe0 con 0x7f01bc15e110 2024-01-15T14:35:37.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.844+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01bc069020 con 0x7f01bc15e110 2024-01-15T14:35:37.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.845+0000 7f01b2ffd700 1 -- 172.21.15.77:0/1531990584 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f01b4058600 con 0x7f01bc15e110 2024-01-15T14:35:37.846 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:37.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f01a006b380 msgr2=0x7f01a006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f01a006b380 0x7f01a006d840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f01ac000c00 tx=0x7f01ac008030 comp rx=0 tx=0).stop 2024-01-15T14:35:37.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 msgr2=0x7f01bc15a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc15a760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f01b400cef0 tx=0x7f01b400a340 comp rx=0 tx=0).stop 2024-01-15T14:35:37.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 shutdown_connections 2024-01-15T14:35:37.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f01a006b380 0x7f01a006d840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f01bc06e420 0x7f01bc1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01bc15e110 0x7f01bc15a760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 --2- 172.21.15.77:0/1531990584 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f01bc0afb30 0x7f01bc157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 >> 172.21.15.77:0/1531990584 conn(0x7f01bc0b3250 msgr2=0x7f01bc06bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:37.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 shutdown_connections 2024-01-15T14:35:37.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.847+0000 7f01c432d700 1 -- 172.21.15.77:0/1531990584 wait complete. 2024-01-15T14:35:37.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.983+0000 7f116cace700 1 Processor -- start 2024-01-15T14:35:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.983+0000 7f116cace700 1 -- start start 2024-01-15T14:35:37.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116cace700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 0x7f116815b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116cace700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116806eeb0 con 0x7f116815b240 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116cace700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116806f190 con 0x7f116815bb60 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116cace700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116806f470 con 0x7f116806c2e0 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f116815b240 0x7f116806e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43588/0 (socket says 172.21.15.77:43588) 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116659c700 1 -- 172.21.15.77:0/3190923384 learned_addr learned my addr 172.21.15.77:0/3190923384 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f1165d9b700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 0x7f116815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296858823 0 0) 0x7f116806eeb0 con 0x7f116815b240 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11500035b0 con 0x7f116815b240 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3212143214 0 0) 0x7f11500035b0 con 0x7f116815b240 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 msgr2=0x7f116815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 0x7f116815b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 >> v1:172.21.15.159:6789/0 conn(0x7f116815bb60 legacy=0x7f11681647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116806f750 con 0x7f116815b240 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.984+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f1158001670 con 0x7f116815b240 2024-01-15T14:35:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 >> v1:172.21.15.77:6789/0 conn(0x7f116815b240 legacy=0x7f116806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 0x7f1150003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f1150004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1150004d90 0x7f1150005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116806f470 con 0x7f1150003f70 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115000ba70 con 0x7f1150004d90 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115000bc30 con 0x7f116806c2e0 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f1150004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116659c700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 0x7f1150003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1166d9d700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1150004d90 0x7f1150005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 -- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 msgr2=0x7f1150003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f116806c2e0 0x7f1150003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 -- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1150004d90 msgr2=0x7f1150005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1150004d90 0x7f1150005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 -- 172.21.15.77:0/3190923384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500035b0 con 0x7f1150003f70 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f1165d9b700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f1150004850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f115c009b60 tx=0x7f1150005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b531d677e12a2dbd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00e6c0 con 0x7f1150003f70 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c00e820 con 0x7f1150003f70 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.985+0000 7f116559a700 1 -- 172.21.15.77:0/3190923384 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00eb90 con 0x7f1150003f70 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f116cace700 1 -- 172.21.15.77:0/3190923384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 msgr2=0x7f1150004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f116cace700 1 --2- 172.21.15.77:0/3190923384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f1150004850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f115c009b60 tx=0x7f1150005880 comp rx=0 tx=0).stop 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f1166d9d700 1 -- 172.21.15.77:0/3190923384 reap_dead start 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f116cace700 1 -- 172.21.15.77:0/3190923384 shutdown_connections 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f116cace700 1 -- 172.21.15.77:0/3190923384 >> 172.21.15.77:0/3190923384 conn(0x7f11680d0c00 msgr2=0x7f11680d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f116cace700 1 -- 172.21.15.77:0/3190923384 shutdown_connections 2024-01-15T14:35:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.986+0000 7f116cace700 1 -- 172.21.15.77:0/3190923384 wait complete. 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.987+0000 7f116cace700 1 Processor -- start 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.987+0000 7f116cace700 1 -- start start 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116cace700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f116814ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116cace700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1168072590 0x7f11680b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116cace700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11680b21c0 0x7f1168159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116cace700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116809d1a0 con 0x7f1150003f70 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116cace700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680b2660 con 0x7f1168072590 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116cace700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116809bb30 con 0x7f11680b21c0 2024-01-15T14:35:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f116814ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f116814ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57912/0 (socket says 172.21.15.77:57912) 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 -- 172.21.15.77:0/811208681 learned_addr learned my addr 172.21.15.77:0/811208681 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f1166d9d700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11680b21c0 0x7f1168159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f1165d9b700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1168072590 0x7f11680b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 -- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11680b21c0 msgr2=0x7f1168159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11680b21c0 0x7f1168159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 -- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1168072590 msgr2=0x7f11680b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1168072590 0x7f11680b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 -- 172.21.15.77:0/811208681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116809be10 con 0x7f1150003f70 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.988+0000 7f116659c700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f116814ec40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1158007290 tx=0x7f11580075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.989+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115800f600 con 0x7f1150003f70 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.989+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115800f7a0 con 0x7f1150003f70 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.989+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116809c0f0 con 0x7f1150003f70 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.989+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115800fb50 con 0x7f1150003f70 2024-01-15T14:35:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.989+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116815a120 con 0x7f1150003f70 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.990+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116806eb50 con 0x7f1150003f70 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.990+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f115800fcb0 con 0x7f1150003f70 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.990+0000 7f1156ffd700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f114806b350 0x7f114806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.990+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1158089ad0 con 0x7f1150003f70 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.991+0000 7f1165d9b700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f114806b350 0x7f114806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.991+0000 7f1165d9b700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f114806b350 0x7f114806d810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f115c0090b0 tx=0x7f115c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:37.995+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f116806eb50 con 0x7f1150003f70 2024-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.183+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f116809bc70 con 0x7f1150003f70 2024-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.184+0000 7f1156ffd700 1 -- 172.21.15.77:0/811208681 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f116809bc70 con 0x7f1150003f70 2024-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.185+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f114806b350 msgr2=0x7f114806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.185+0000 7f116cace700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f114806b350 0x7f114806d810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f115c0090b0 tx=0x7f115c008030 comp rx=0 tx=0).stop 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 msgr2=0x7f116814ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f116814ec40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1158007290 tx=0x7f11580075a0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 shutdown_connections 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f114806b350 0x7f114806d810 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f11680b21c0 0x7f1168159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1150003f70 0x7f116814ec40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 --2- 172.21.15.77:0/811208681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1168072590 0x7f11680b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 >> 172.21.15.77:0/811208681 conn(0x7f11680d0c00 msgr2=0x7f1168153230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 shutdown_connections 2024-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.186+0000 7f116cace700 1 -- 172.21.15.77:0/811208681 wait complete. 2024-01-15T14:35:38.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T14:35:38.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 Processor -- start 2024-01-15T14:35:38.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 -- start start 2024-01-15T14:35:38.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe29815e1e0 0x7fe2980d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d3a00 0x7fe2980d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2980d20b0 con 0x7fe2980d0db0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2980d3150 con 0x7fe29815e1e0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.321+0000 7fe29d374700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980d1240 con 0x7fe2980d3a00 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe296ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe29815e1e0 0x7fe2980d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe2977fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe2980d0db0 0x7fe29806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43592/0 (socket says 172.21.15.77:43592) 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe2977fe700 1 -- 172.21.15.77:0/2547449881 learned_addr learned my addr 172.21.15.77:0/2547449881 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe296ffd700 1 -- 172.21.15.77:0/2547449881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d3a00 msgr2=0x7fe2980d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe2967fc700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d3a00 0x7fe2980d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe296ffd700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d3a00 0x7fe2980d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe296ffd700 1 -- 172.21.15.77:0/2547449881 >> v1:172.21.15.77:6789/0 conn(0x7fe2980d0db0 legacy=0x7fe29806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe295ffb700 1 -- 172.21.15.77:0/2547449881 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325417166 0 0) 0x7fe2980d20b0 con 0x7fe2980d0db0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe296ffd700 1 -- 172.21.15.77:0/2547449881 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2980d2390 con 0x7fe29815e1e0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe2967fc700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d3a00 0x7fe2980d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe296ffd700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe29815e1e0 0x7fe2980d34c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe288009840 tx=0x7fe288009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c142fcfd24e96c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe295ffb700 1 -- 172.21.15.77:0/2547449881 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe288011830 con 0x7fe29815e1e0 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe295ffb700 1 -- 172.21.15.77:0/2547449881 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe288011990 con 0x7fe29815e1e0 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.322+0000 7fe295ffb700 1 -- 172.21.15.77:0/2547449881 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe288011d20 con 0x7fe29815e1e0 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 -- 172.21.15.77:0/2547449881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe29815e1e0 msgr2=0x7fe2980d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe29815e1e0 0x7fe2980d34c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe288009840 tx=0x7fe288009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 -- 172.21.15.77:0/2547449881 shutdown_connections 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe29815e1e0 0x7fe2980d34c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 --2- 172.21.15.77:0/2547449881 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d3a00 0x7fe2980d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 -- 172.21.15.77:0/2547449881 >> 172.21.15.77:0/2547449881 conn(0x7fe2980b5b50 msgr2=0x7fe2980b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 -- 172.21.15.77:0/2547449881 shutdown_connections 2024-01-15T14:35:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.323+0000 7fe29d374700 1 -- 172.21.15.77:0/2547449881 wait complete. 2024-01-15T14:35:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.324+0000 7fe29d374700 1 Processor -- start 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.324+0000 7fe29d374700 1 -- start start 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe29d374700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d0db0 0x7fe2980cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe29d374700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 0x7fe29815e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe29d374700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe29815e1e0 0x7fe298155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe29d374700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980cc8d0 con 0x7fe2980d3a00 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe29d374700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe298155030 con 0x7fe29815e1e0 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe29d374700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980cc350 con 0x7fe2980d0db0 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 0x7fe29815e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 0x7fe29815e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57926/0 (socket says 172.21.15.77:57926) 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 -- 172.21.15.77:0/1138659278 learned_addr learned my addr 172.21.15.77:0/1138659278 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe296ffd700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d0db0 0x7fe2980cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2977fe700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe29815e1e0 0x7fe298155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 -- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d0db0 msgr2=0x7fe2980cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d0db0 0x7fe2980cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 -- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe29815e1e0 msgr2=0x7fe298155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe29815e1e0 0x7fe298155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2967fc700 1 -- 172.21.15.77:0/1138659278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2880094a0 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.325+0000 7fe2977fe700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe29815e1e0 0x7fe298155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.326+0000 7fe2967fc700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 0x7fe29815e740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe28000c750 tx=0x7fe28000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.326+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe280011800 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.326+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe280011960 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.326+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2980cc5f0 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.326+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe280011cd0 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.326+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2981555b0 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.327+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2980751e0 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.327+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe280021350 con 0x7fe2980d3a00 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.327+0000 7fe28f7fe700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe27806b350 0x7fe27806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.328+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe280089a50 con 0x7fe2980d3a00 2024-01-15T14:35:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.328+0000 7fe296ffd700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe27806b350 0x7fe27806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.328+0000 7fe296ffd700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe27806b350 0x7fe27806d810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe2880090b0 tx=0x7fe28801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.332+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe280054950 con 0x7fe2980d3a00 2024-01-15T14:35:38.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.537+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe2980cb1e0 con 0x7fe2980d3a00 2024-01-15T14:35:38.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.538+0000 7fe28f7fe700 1 -- 172.21.15.77:0/1138659278 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe28001e080 con 0x7fe2980d3a00 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe27806b350 msgr2=0x7fe27806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe27806b350 0x7fe27806d810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe2880090b0 tx=0x7fe28801c040 comp rx=0 tx=0).stop 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 msgr2=0x7fe29815e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 0x7fe29815e740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe28000c750 tx=0x7fe28000cb10 comp rx=0 tx=0).stop 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 shutdown_connections 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe27806b350 0x7fe27806d810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe2980d0db0 0x7fe2980cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe29815e1e0 0x7fe298155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 --2- 172.21.15.77:0/1138659278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2980d3a00 0x7fe29815e740 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 >> 172.21.15.77:0/1138659278 conn(0x7fe2980b5b50 msgr2=0x7fe2980b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 shutdown_connections 2024-01-15T14:35:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.540+0000 7fe29d374700 1 -- 172.21.15.77:0/1138659278 wait complete. 2024-01-15T14:35:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-01-15T14:35:38.553 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-15T14:35:38.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:38.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-01-15T14:35:38.590 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:35:38.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:38.590 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T14:35:38.658 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:38.658 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:38.680 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:38.680 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:38.741 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:38.741 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:38.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 Processor -- start 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 -- start start 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709006e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f709006ea50 0x7f709015e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f709015ec80 0x7f709015b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709006aff0 con 0x7f709006ea50 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709015f0c0 con 0x7f709015ec80 2024-01-15T14:35:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f7096f14700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709015f510 con 0x7f70901445b0 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.924+0000 7f708ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f709006ea50 0x7f709015e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f709006ea50 0x7f709015e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57942/0 (socket says 172.21.15.77:57942) 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 -- 172.21.15.77:0/1433093274 learned_addr learned my addr 172.21.15.77:0/1433093274 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 -- 172.21.15.77:0/1433093274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 msgr2=0x7f709006e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 --2- 172.21.15.77:0/1433093274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709006e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 -- 172.21.15.77:0/1433093274 >> v2:172.21.15.159:3300/0 conn(0x7f709015ec80 msgr2=0x7f709015b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 --2- 172.21.15.77:0/1433093274 >> v2:172.21.15.159:3300/0 conn(0x7f709015ec80 0x7f709015b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 -- 172.21.15.77:0/1433093274 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709006f060 con 0x7f709006ea50 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708ffff700 1 --2- 172.21.15.77:0/1433093274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709015e740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f709006e060 tx=0x7f707c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd954fe6b50f1d6e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708f7fe700 1 -- 172.21.15.77:0/1433093274 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c00b870 con 0x7f709006ea50 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708f7fe700 1 -- 172.21.15.77:0/1433093274 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c00b9d0 con 0x7f709006ea50 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.925+0000 7f708f7fe700 1 -- 172.21.15.77:0/1433093274 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c00bd60 con 0x7f709006ea50 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 -- 172.21.15.77:0/1433093274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 msgr2=0x7f709015e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 --2- 172.21.15.77:0/1433093274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709015e740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f709006e060 tx=0x7f707c009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 -- 172.21.15.77:0/1433093274 shutdown_connections 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 --2- 172.21.15.77:0/1433093274 >> v2:172.21.15.159:3300/0 conn(0x7f709015ec80 0x7f709015b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 --2- 172.21.15.77:0/1433093274 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709006e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 --2- 172.21.15.77:0/1433093274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709015e740 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 -- 172.21.15.77:0/1433093274 >> 172.21.15.77:0/1433093274 conn(0x7f70900d0ac0 msgr2=0x7f70900d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 -- 172.21.15.77:0/1433093274 shutdown_connections 2024-01-15T14:35:38.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.926+0000 7f7096f14700 1 -- 172.21.15.77:0/1433093274 wait complete. 2024-01-15T14:35:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.927+0000 7f7096f14700 1 Processor -- start 2024-01-15T14:35:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.927+0000 7f7096f14700 1 -- start start 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7096f14700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709013f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7096f14700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709013d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7096f14700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f709015ec80 0x7f7090139900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7096f14700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709013d850 con 0x7f709006ea50 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7096f14700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709013a4d0 con 0x7f709015ec80 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7096f14700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709013dc20 con 0x7f70901445b0 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7094cb0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709013f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7094cb0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709013f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57948/0 (socket says 172.21.15.77:57948) 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.928+0000 7f7094cb0700 1 -- 172.21.15.77:0/3226965918 learned_addr learned my addr 172.21.15.77:0/3226965918 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f70954b1700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f709015ec80 0x7f7090139900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f708ffff700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709013d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7094cb0700 1 -- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 msgr2=0x7f709013d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7094cb0700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709013d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7094cb0700 1 -- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f709015ec80 msgr2=0x7f7090139900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7094cb0700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f709015ec80 0x7f7090139900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7094cb0700 1 -- 172.21.15.77:0/3226965918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c009550 con 0x7f709006ea50 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7094cb0700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709013f7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f708400b530 tx=0x7f708400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084011830 con 0x7f709006ea50 2024-01-15T14:35:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70840119b0 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709013a770 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084011d60 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.929+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709013df00 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.930+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709013d990 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.930+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f708401a400 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.931+0000 7f708d7fa700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f707406b350 0x7f707406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.931+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7084089c10 con 0x7f709006ea50 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.931+0000 7f708ffff700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f707406b350 0x7f707406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.931+0000 7f708ffff700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f707406b350 0x7f707406d810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f707c000c00 tx=0x7f707c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:38.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:38.935+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7084054a80 con 0x7f709006ea50 2024-01-15T14:35:39.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.127+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7090067f60 con 0x7f709006ea50 2024-01-15T14:35:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.128+0000 7f708d7fa700 1 -- 172.21.15.77:0/3226965918 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f7084015020 con 0x7f709006ea50 2024-01-15T14:35:39.130 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:39.130 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f707406b350 msgr2=0x7f707406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f707406b350 0x7f707406d810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f707c000c00 tx=0x7f707c008030 comp rx=0 tx=0).stop 2024-01-15T14:35:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 msgr2=0x7f709013f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709013f7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f708400b530 tx=0x7f708400b8f0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 shutdown_connections 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f707406b350 0x7f707406d810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f709015ec80 0x7f7090139900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f709006ea50 0x7f709013f7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 --2- 172.21.15.77:0/3226965918 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f70901445b0 0x7f709013d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 >> 172.21.15.77:0/3226965918 conn(0x7f70900d0ac0 msgr2=0x7f709013e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 shutdown_connections 2024-01-15T14:35:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.131+0000 7f7096f14700 1 -- 172.21.15.77:0/3226965918 wait complete. 2024-01-15T14:35:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:39.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:39.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.269+0000 7fc76c0a1700 1 Processor -- start 2024-01-15T14:35:39.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.270+0000 7fc76c0a1700 1 -- start start 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.270+0000 7fc76c0a1700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc76415f8d0 0x7fc7640b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.270+0000 7fc76c0a1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640b3860 con 0x7fc7640d03a0 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.270+0000 7fc76c0a1700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640b2520 con 0x7fc7640cffc0 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.270+0000 7fc76c0a1700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7640b2800 con 0x7fc76415f8d0 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76963c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc76415f8d0 0x7fc7640b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76a63e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc7640d03a0 0x7fc7640ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47190/0 (socket says 172.21.15.77:47190) 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76a63e700 1 -- 172.21.15.77:0/1409035777 learned_addr learned my addr 172.21.15.77:0/1409035777 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:39.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76963c700 1 -- 172.21.15.77:0/1409035777 >> v1:172.21.15.159:6789/0 conn(0x7fc7640cffc0 legacy=0x7fc76415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76963c700 1 -- 172.21.15.77:0/1409035777 >> v1:172.21.15.77:6789/0 conn(0x7fc7640d03a0 legacy=0x7fc7640ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76963c700 1 -- 172.21.15.77:0/1409035777 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640b2ae0 con 0x7fc76415f8d0 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc76963c700 1 --2- 172.21.15.77:0/1409035777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76415f8d0 0x7fc7640b3320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc754009840 tx=0x7fc754009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a202e6aa521513 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc768e3b700 1 -- 172.21.15.77:0/1409035777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754004230 con 0x7fc76415f8d0 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc768e3b700 1 -- 172.21.15.77:0/1409035777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc754004390 con 0x7fc76415f8d0 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.271+0000 7fc768e3b700 1 -- 172.21.15.77:0/1409035777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540036a0 con 0x7fc76415f8d0 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1409035777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76415f8d0 msgr2=0x7fc7640b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1409035777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76415f8d0 0x7fc7640b3320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc754009840 tx=0x7fc754009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1409035777 shutdown_connections 2024-01-15T14:35:39.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1409035777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76415f8d0 0x7fc7640b3320 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1409035777 >> 172.21.15.77:0/1409035777 conn(0x7fc7640d1e50 msgr2=0x7fc7640d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:39.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1409035777 shutdown_connections 2024-01-15T14:35:39.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.272+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1409035777 wait complete. 2024-01-15T14:35:39.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.273+0000 7fc76c0a1700 1 Processor -- start 2024-01-15T14:35:39.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.273+0000 7fc76c0a1700 1 -- start start 2024-01-15T14:35:39.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7640cffc0 0x7fc7641f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 0x7fc7641f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76415f8d0 0x7fc7640af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7641f5ba0 con 0x7fc7640d03a0 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640c7040 con 0x7fc7640cffc0 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76414dfc0 con 0x7fc76415f8d0 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 0x7fc7641f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 0x7fc7641f0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40338/0 (socket says 172.21.15.77:40338) 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 -- 172.21.15.77:0/1044508951 learned_addr learned my addr 172.21.15.77:0/1044508951 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc769e3d700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7640cffc0 0x7fc7641f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76a63e700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76415f8d0 0x7fc7640af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 -- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76415f8d0 msgr2=0x7fc7640af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76415f8d0 0x7fc7640af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 -- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7640cffc0 msgr2=0x7fc7641f5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7640cffc0 0x7fc7641f5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 -- 172.21.15.77:0/1044508951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7540094a0 con 0x7fc7640d03a0 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76a63e700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76415f8d0 0x7fc7640af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc769e3d700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7640cffc0 0x7fc7641f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76963c700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 0x7fc7641f0cf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc7540090b0 tx=0x7fc754004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754003e70 con 0x7fc7640d03a0 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75401f070 con 0x7fc7640d03a0 2024-01-15T14:35:39.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76414e2a0 con 0x7fc7640d03a0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754011890 con 0x7fc7640d03a0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.274+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76414e860 con 0x7fc7640d03a0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.275+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7640c71d0 con 0x7fc7640d03a0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.276+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc754021030 con 0x7fc7640d03a0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.276+0000 7fc75a7fc700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc74806b350 0x7fc74806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.276+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc75408b210 con 0x7fc7640d03a0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.276+0000 7fc769e3d700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc74806b350 0x7fc74806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.277+0000 7fc769e3d700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc74806b350 0x7fc74806d810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc7600017d0 tx=0x7fc760009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.281+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc754056070 con 0x7fc7640d03a0 2024-01-15T14:35:39.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.469+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc764069020 con 0x7fc7640d03a0 2024-01-15T14:35:39.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.469+0000 7fc75a7fc700 1 -- 172.21.15.77:0/1044508951 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc754059690 con 0x7fc7640d03a0 2024-01-15T14:35:39.470 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc74806b350 msgr2=0x7fc74806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc74806b350 0x7fc74806d810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc7600017d0 tx=0x7fc760009040 comp rx=0 tx=0).stop 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 msgr2=0x7fc7641f0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 0x7fc7641f0cf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc7540090b0 tx=0x7fc754004750 comp rx=0 tx=0).stop 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 shutdown_connections 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc74806b350 0x7fc74806d810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc76415f8d0 0x7fc7640af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc7640cffc0 0x7fc7641f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 --2- 172.21.15.77:0/1044508951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7640d03a0 0x7fc7641f0cf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 >> 172.21.15.77:0/1044508951 conn(0x7fc7640d1e50 msgr2=0x7fc7640d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:39.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 shutdown_connections 2024-01-15T14:35:39.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.471+0000 7fc76c0a1700 1 -- 172.21.15.77:0/1044508951 wait complete. 2024-01-15T14:35:39.485 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:39.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.608+0000 7f7173d63700 1 Processor -- start 2024-01-15T14:35:39.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.608+0000 7f7173d63700 1 -- start start 2024-01-15T14:35:39.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7173d63700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f716c161690 0x7f716c0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7173d63700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f716c09b110 0x7f716c0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7173d63700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c0d0a70 con 0x7f716c161a70 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7173d63700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c0d1c10 con 0x7f716c161690 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7173d63700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c0b2ab0 con 0x7f716c09b110 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7171aff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f716c161690 0x7f716c0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f71712fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f716c161a70 0x7f716c09aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47194/0 (socket says 172.21.15.77:47194) 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f71712fe700 1 -- 172.21.15.77:0/1942188294 learned_addr learned my addr 172.21.15.77:0/1942188294 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7172300700 1 --2- 172.21.15.77:0/1942188294 >> v2:172.21.15.159:3300/0 conn(0x7f716c09b110 0x7f716c0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7171aff700 1 -- 172.21.15.77:0/1942188294 >> v2:172.21.15.159:3300/0 conn(0x7f716c09b110 msgr2=0x7f716c0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7171aff700 1 --2- 172.21.15.77:0/1942188294 >> v2:172.21.15.159:3300/0 conn(0x7f716c09b110 0x7f716c0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7171aff700 1 -- 172.21.15.77:0/1942188294 >> v1:172.21.15.77:6789/0 conn(0x7f716c161a70 legacy=0x7f716c09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7170afd700 1 -- 172.21.15.77:0/1942188294 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959065401 0 0) 0x7f716c0d0a70 con 0x7f716c161a70 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.609+0000 7f7171aff700 1 -- 172.21.15.77:0/1942188294 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0b2d90 con 0x7f716c161690 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7171aff700 1 --2- 172.21.15.77:0/1942188294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c0d0f20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f716c0ce460 tx=0x7f715c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7b73da53f8a97a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7170afd700 1 -- 172.21.15.77:0/1942188294 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c011830 con 0x7f716c161690 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7170afd700 1 -- 172.21.15.77:0/1942188294 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c011990 con 0x7f716c161690 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7170afd700 1 -- 172.21.15.77:0/1942188294 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c011d20 con 0x7f716c161690 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 -- 172.21.15.77:0/1942188294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 msgr2=0x7f716c0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 --2- 172.21.15.77:0/1942188294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c0d0f20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f716c0ce460 tx=0x7f715c009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 -- 172.21.15.77:0/1942188294 shutdown_connections 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 --2- 172.21.15.77:0/1942188294 >> v2:172.21.15.159:3300/0 conn(0x7f716c09b110 0x7f716c0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 --2- 172.21.15.77:0/1942188294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c0d0f20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 -- 172.21.15.77:0/1942188294 >> 172.21.15.77:0/1942188294 conn(0x7f716c0d1e50 msgr2=0x7f716c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 -- 172.21.15.77:0/1942188294 shutdown_connections 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.610+0000 7f7173d63700 1 -- 172.21.15.77:0/1942188294 wait complete. 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 Processor -- start 2024-01-15T14:35:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 -- start start 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f716c09b110 0x7f716c0c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f716c161a70 0x7f716c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0b7a30 con 0x7f716c161690 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0c88a0 con 0x7f716c09b110 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f7173d63700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0c85c0 con 0x7f716c161a70 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f71712fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f71712fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40356/0 (socket says 172.21.15.77:40356) 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.612+0000 7f71712fe700 1 -- 172.21.15.77:0/4022806622 learned_addr learned my addr 172.21.15.77:0/4022806622 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f7171aff700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f716c09b110 0x7f716c0c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f7172300700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f716c161a70 0x7f716c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71712fe700 1 -- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f716c161a70 msgr2=0x7f716c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71712fe700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f716c161a70 0x7f716c0707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71712fe700 1 -- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f716c09b110 msgr2=0x7f716c0c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71712fe700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f716c09b110 0x7f716c0c5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71712fe700 1 -- 172.21.15.77:0/4022806622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c0094a0 con 0x7f716c161690 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f7171aff700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f716c09b110 0x7f716c0c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71712fe700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c070290 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f71680011e0 tx=0x7f716800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168004c80 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7168004e00 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c147160 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168007310 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.613+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c147720 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.614+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c147940 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.614+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7168007470 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.615+0000 7f71627fc700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f715006b350 0x7f715006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.615+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7168089940 con 0x7f716c161690 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.615+0000 7f7171aff700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f715006b350 0x7f715006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.615+0000 7f7171aff700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f715006b350 0x7f715006d810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f715c0090b0 tx=0x7f715c01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.619+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71680547b0 con 0x7f716c161690 2024-01-15T14:35:39.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.807+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f716c14b340 con 0x7f716c161690 2024-01-15T14:35:39.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.807+0000 7f71627fc700 1 -- 172.21.15.77:0/4022806622 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7168057dd0 con 0x7f716c161690 2024-01-15T14:35:39.809 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:39.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f715006b350 msgr2=0x7f715006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f715006b350 0x7f715006d810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f715c0090b0 tx=0x7f715c01c040 comp rx=0 tx=0).stop 2024-01-15T14:35:39.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 msgr2=0x7f716c070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c070290 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f71680011e0 tx=0x7f716800cae0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 shutdown_connections 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f715006b350 0x7f715006d810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f716c161a70 0x7f716c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f716c09b110 0x7f716c0c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 --2- 172.21.15.77:0/4022806622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f716c161690 0x7f716c070290 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 >> 172.21.15.77:0/4022806622 conn(0x7f716c0d1e50 msgr2=0x7f716c06f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 shutdown_connections 2024-01-15T14:35:39.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.809+0000 7f7173d63700 1 -- 172.21.15.77:0/4022806622 wait complete. 2024-01-15T14:35:39.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.946+0000 7fd089a18700 1 Processor -- start 2024-01-15T14:35:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.946+0000 7fd089a18700 1 -- start start 2024-01-15T14:35:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd089a18700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd08415df60 0x7fd08415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd089a18700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840b3f60 con 0x7fd0840b3470 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd089a18700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08415e8f0 con 0x7fd084151620 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd089a18700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08406d190 con 0x7fd08415df60 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd0837fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd08415df60 0x7fd08415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd0827fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd0840b3470 0x7fd084070b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47208/0 (socket says 172.21.15.77:47208) 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd0827fc700 1 -- 172.21.15.77:0/1776601362 learned_addr learned my addr 172.21.15.77:0/1776601362 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd0837fe700 1 -- 172.21.15.77:0/1776601362 >> v1:172.21.15.159:6789/0 conn(0x7fd084151620 legacy=0x7fd0840cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd0837fe700 1 -- 172.21.15.77:0/1776601362 >> v1:172.21.15.77:6789/0 conn(0x7fd0840b3470 legacy=0x7fd084070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd0837fe700 1 -- 172.21.15.77:0/1776601362 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08406d750 con 0x7fd08415df60 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.947+0000 7fd081ffb700 1 -- 172.21.15.77:0/1776601362 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952709008 0 0) 0x7fd0840b3f60 con 0x7fd0840b3470 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd0837fe700 1 --2- 172.21.15.77:0/1776601362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd08415e3b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009d90 tx=0x7fd07c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6912ba2cf4b08c08 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd081ffb700 1 -- 172.21.15.77:0/1776601362 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c00e070 con 0x7fd08415df60 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd081ffb700 1 -- 172.21.15.77:0/1776601362 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c00bbd0 con 0x7fd08415df60 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd081ffb700 1 -- 172.21.15.77:0/1776601362 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c012430 con 0x7fd08415df60 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 -- 172.21.15.77:0/1776601362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 msgr2=0x7fd08415e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 --2- 172.21.15.77:0/1776601362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd08415e3b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009d90 tx=0x7fd07c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 -- 172.21.15.77:0/1776601362 shutdown_connections 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 --2- 172.21.15.77:0/1776601362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd08415e3b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 -- 172.21.15.77:0/1776601362 >> 172.21.15.77:0/1776601362 conn(0x7fd08406b410 msgr2=0x7fd08406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 -- 172.21.15.77:0/1776601362 shutdown_connections 2024-01-15T14:35:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.948+0000 7fd089a18700 1 -- 172.21.15.77:0/1776601362 wait complete. 2024-01-15T14:35:39.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.949+0000 7fd089a18700 1 Processor -- start 2024-01-15T14:35:39.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 -- start start 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0840b3470 0x7fd084142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd084151620 0x7fd0840b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd084141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084142410 con 0x7fd08415df60 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0841481d0 con 0x7fd084151620 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd089a18700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0841484b0 con 0x7fd0840b3470 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd084141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd084141440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40376/0 (socket says 172.21.15.77:40376) 2024-01-15T14:35:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.950+0000 7fd0837fe700 1 -- 172.21.15.77:0/2602889842 learned_addr learned my addr 172.21.15.77:0/2602889842 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0827fc700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd084151620 0x7fd0840b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd082ffd700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0840b3470 0x7fd084142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0837fe700 1 -- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0840b3470 msgr2=0x7fd084142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0837fe700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0840b3470 0x7fd084142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0837fe700 1 -- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd084151620 msgr2=0x7fd0840b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0837fe700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd084151620 0x7fd0840b0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0837fe700 1 -- 172.21.15.77:0/2602889842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c009a70 con 0x7fd08415df60 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd0837fe700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd084141440 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd07c014040 tx=0x7fd07c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c01b920 con 0x7fd08415df60 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c020070 con 0x7fd08415df60 2024-01-15T14:35:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd084148790 con 0x7fd08415df60 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c01bcc0 con 0x7fd08415df60 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.951+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd084148d50 con 0x7fd08415df60 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.952+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd084148f40 con 0x7fd08415df60 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.952+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd07c026400 con 0x7fd08415df60 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.953+0000 7fd06bfff700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd07006b350 0x7fd07006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.953+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd07c08ba80 con 0x7fd08415df60 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.953+0000 7fd082ffd700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd07006b350 0x7fd07006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:39.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.953+0000 7fd082ffd700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd07006b350 0x7fd07006d810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd078000f60 tx=0x7fd078009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:39.957+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd07c05a410 con 0x7fd08415df60 2024-01-15T14:35:40.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.147+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0840711b0 con 0x7fd08415df60 2024-01-15T14:35:40.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.148+0000 7fd06bfff700 1 -- 172.21.15.77:0/2602889842 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd07c0220a0 con 0x7fd08415df60 2024-01-15T14:35:40.149 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:40.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd07006b350 msgr2=0x7fd07006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd07006b350 0x7fd07006d810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd078000f60 tx=0x7fd078009040 comp rx=0 tx=0).stop 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 msgr2=0x7fd084141440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd084141440 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd07c014040 tx=0x7fd07c004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 shutdown_connections 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd07006b350 0x7fd07006d810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd0840b3470 0x7fd084142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd08415df60 0x7fd084141440 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 --2- 172.21.15.77:0/2602889842 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd084151620 0x7fd0840b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 >> 172.21.15.77:0/2602889842 conn(0x7fd08406b410 msgr2=0x7fd08406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 shutdown_connections 2024-01-15T14:35:40.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.150+0000 7fd089a18700 1 -- 172.21.15.77:0/2602889842 wait complete. 2024-01-15T14:35:40.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T14:35:40.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.285+0000 7f771fc76700 1 Processor -- start 2024-01-15T14:35:40.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.285+0000 7f771fc76700 1 -- start start 2024-01-15T14:35:40.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771fc76700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f77181473a0 0x7f771815e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771fc76700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f771806c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771fc76700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771806c9a0 con 0x7f771806dc20 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771fc76700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771815eb00 con 0x7f77181473a0 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771fc76700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771815ede0 con 0x7f7718147780 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771da12700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f77181473a0 0x7f771815e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771e213700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f771806dc20 0x7f7718073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47222/0 (socket says 172.21.15.77:47222) 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771e213700 1 -- 172.21.15.77:0/1625030846 learned_addr learned my addr 172.21.15.77:0/1625030846 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:40.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771da12700 1 -- 172.21.15.77:0/1625030846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 msgr2=0x7f771806c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:40.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771da12700 1 --2- 172.21.15.77:0/1625030846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f771806c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771da12700 1 -- 172.21.15.77:0/1625030846 >> v1:172.21.15.77:6789/0 conn(0x7f771806dc20 legacy=0x7f7718073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.286+0000 7f771da12700 1 -- 172.21.15.77:0/1625030846 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771815f0c0 con 0x7f77181473a0 2024-01-15T14:35:40.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771da12700 1 --2- 172.21.15.77:0/1625030846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f771815e4b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7708009840 tx=0x7f7708009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc57a303ca0a03 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771ca10700 1 -- 172.21.15.77:0/1625030846 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708011830 con 0x7f77181473a0 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771ca10700 1 -- 172.21.15.77:0/1625030846 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708011990 con 0x7f77181473a0 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771ca10700 1 -- 172.21.15.77:0/1625030846 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708011d20 con 0x7f77181473a0 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 -- 172.21.15.77:0/1625030846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 msgr2=0x7f771815e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 --2- 172.21.15.77:0/1625030846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f771815e4b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7708009840 tx=0x7f7708009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 -- 172.21.15.77:0/1625030846 shutdown_connections 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 --2- 172.21.15.77:0/1625030846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f771815e4b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 --2- 172.21.15.77:0/1625030846 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f771806c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 -- 172.21.15.77:0/1625030846 >> 172.21.15.77:0/1625030846 conn(0x7f77180b3130 msgr2=0x7f77180d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:40.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 -- 172.21.15.77:0/1625030846 shutdown_connections 2024-01-15T14:35:40.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.287+0000 7f771fc76700 1 -- 172.21.15.77:0/1625030846 wait complete. 2024-01-15T14:35:40.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.288+0000 7f771fc76700 1 Processor -- start 2024-01-15T14:35:40.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.288+0000 7f771fc76700 1 -- start start 2024-01-15T14:35:40.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771fc76700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771806dc20 0x7f771815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771fc76700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f7718141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771fc76700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f7718145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771fc76700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771815db40 con 0x7f77181473a0 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771fc76700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77181f4630 con 0x7f771806dc20 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771fc76700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77181f4910 con 0x7f7718147780 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771d211700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f7718141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771d211700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f7718141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40396/0 (socket says 172.21.15.77:40396) 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.289+0000 7f771d211700 1 -- 172.21.15.77:0/457611813 learned_addr learned my addr 172.21.15.77:0/457611813 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771da12700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771806dc20 0x7f771815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771e213700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f7718145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771d211700 1 -- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 msgr2=0x7f7718145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771d211700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f7718145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771d211700 1 -- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771806dc20 msgr2=0x7f771815d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771d211700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771806dc20 0x7f771815d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771d211700 1 -- 172.21.15.77:0/457611813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080094a0 con 0x7f77181473a0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771e213700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f7718145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771d211700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f7718141e10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f771400b450 tx=0x7f771400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714011830 con 0x7f77181473a0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7714011990 con 0x7f77181473a0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77181f4bf0 con 0x7f77181473a0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714011d40 con 0x7f77181473a0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.290+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77180afa90 con 0x7f77181473a0 2024-01-15T14:35:40.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.291+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7718067f60 con 0x7f77181473a0 2024-01-15T14:35:40.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.291+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f771401a400 con 0x7f77181473a0 2024-01-15T14:35:40.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.292+0000 7f770e7fc700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f76fc06b350 0x7f76fc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.292+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7714089ac0 con 0x7f77181473a0 2024-01-15T14:35:40.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.292+0000 7f771da12700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f76fc06b350 0x7f76fc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.292+0000 7f771da12700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f76fc06b350 0x7f76fc06d810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7708000c00 tx=0x7f7708008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:40.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.296+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77140549c0 con 0x7f77181473a0 2024-01-15T14:35:40.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.504+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f771806c240 con 0x7f77181473a0 2024-01-15T14:35:40.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.504+0000 7f770e7fc700 1 -- 172.21.15.77:0/457611813 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7714089e60 con 0x7f77181473a0 2024-01-15T14:35:40.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f76fc06b350 msgr2=0x7f76fc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f76fc06b350 0x7f76fc06d810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7708000c00 tx=0x7f7708008030 comp rx=0 tx=0).stop 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 msgr2=0x7f7718141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f7718141e10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f771400b450 tx=0x7f771400b810 comp rx=0 tx=0).stop 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 shutdown_connections 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f76fc06b350 0x7f76fc06d810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7718147780 0x7f7718145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f771806dc20 0x7f771815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 --2- 172.21.15.77:0/457611813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77181473a0 0x7f7718141e10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 >> 172.21.15.77:0/457611813 conn(0x7f77180b3130 msgr2=0x7f7718160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:40.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.506+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 shutdown_connections 2024-01-15T14:35:40.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.507+0000 7f771fc76700 1 -- 172.21.15.77:0/457611813 wait complete. 2024-01-15T14:35:40.510 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-01-15T14:35:40.520 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-01-15T14:35:40.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:40.520 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-01-15T14:35:40.565 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:35:40.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:40.566 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T14:35:40.641 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:40.641 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:40.660 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:40.660 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:40.718 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:35:40.718 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:40.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.936+0000 7ff07632a700 1 Processor -- start 2024-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.936+0000 7ff07632a700 1 -- start start 2024-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff07632a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff07015c030 0x7ff0700d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff07632a700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff0700ae910 0x7ff0700d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff07632a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0700cd5d0 con 0x7ff0700cc8c0 2024-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff07632a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0700cd870 con 0x7ff07015c030 2024-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff07632a700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0700cdb50 con 0x7ff0700ae910 2024-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff07015c030 0x7ff0700d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff0748c7700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff0700cc8c0 0x7ff0700ccd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47224/0 (socket says 172.21.15.77:47224) 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff07015c030 0x7ff0700d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40406/0 (socket says 172.21.15.77:40406) 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff0748c7700 1 -- 172.21.15.77:0/3443720819 learned_addr learned my addr 172.21.15.77:0/3443720819 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06f7fe700 1 --2- 172.21.15.77:0/3443720819 >> v2:172.21.15.159:3300/0 conn(0x7ff0700ae910 0x7ff0700d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06ffff700 1 -- 172.21.15.77:0/3443720819 >> v2:172.21.15.159:3300/0 conn(0x7ff0700ae910 msgr2=0x7ff0700d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06effd700 1 -- 172.21.15.77:0/3443720819 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383118072 0 0) 0x7ff0700cd5d0 con 0x7ff0700cc8c0 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06ffff700 1 --2- 172.21.15.77:0/3443720819 >> v2:172.21.15.159:3300/0 conn(0x7ff0700ae910 0x7ff0700d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06ffff700 1 -- 172.21.15.77:0/3443720819 >> v1:172.21.15.77:6789/0 conn(0x7ff0700cc8c0 legacy=0x7ff0700ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06ffff700 1 -- 172.21.15.77:0/3443720819 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700cde30 con 0x7ff07015c030 2024-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.937+0000 7ff06f7fe700 1 --2- 172.21.15.77:0/3443720819 >> v2:172.21.15.159:3300/0 conn(0x7ff0700ae910 0x7ff0700d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff06ffff700 1 --2- 172.21.15.77:0/3443720819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07015c030 0x7ff0700d0070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff0700cf700 tx=0x7ff064009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9624b6d230645a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff06effd700 1 -- 172.21.15.77:0/3443720819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064011830 con 0x7ff07015c030 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff06effd700 1 -- 172.21.15.77:0/3443720819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064011990 con 0x7ff07015c030 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff06effd700 1 -- 172.21.15.77:0/3443720819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064011d20 con 0x7ff07015c030 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 -- 172.21.15.77:0/3443720819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07015c030 msgr2=0x7ff0700d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 --2- 172.21.15.77:0/3443720819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07015c030 0x7ff0700d0070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff0700cf700 tx=0x7ff064009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 -- 172.21.15.77:0/3443720819 shutdown_connections 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 --2- 172.21.15.77:0/3443720819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07015c030 0x7ff0700d0070 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 --2- 172.21.15.77:0/3443720819 >> v2:172.21.15.159:3300/0 conn(0x7ff0700ae910 0x7ff0700d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 -- 172.21.15.77:0/3443720819 >> 172.21.15.77:0/3443720819 conn(0x7ff0700730a0 msgr2=0x7ff07009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 -- 172.21.15.77:0/3443720819 shutdown_connections 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.938+0000 7ff07632a700 1 -- 172.21.15.77:0/3443720819 wait complete. 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.939+0000 7ff07632a700 1 Processor -- start 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 -- start start 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 0x7ff070159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cc8c0 0x7ff0700c4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0700c4790 0x7ff0701f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07006b460 con 0x7ff0700ae910 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07006b740 con 0x7ff0700c4790 2024-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.940+0000 7ff07632a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07006ba20 con 0x7ff0700cc8c0 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 0x7ff070159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 0x7ff070159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40412/0 (socket says 172.21.15.77:40412) 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 -- 172.21.15.77:0/3128039224 learned_addr learned my addr 172.21.15.77:0/3128039224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff0748c7700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0700c4790 0x7ff0701f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06f7fe700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cc8c0 0x7ff0700c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 -- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cc8c0 msgr2=0x7ff0700c4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cc8c0 0x7ff0700c4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 -- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0700c4790 msgr2=0x7ff0701f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0700c4790 0x7ff0701f6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 -- 172.21.15.77:0/3128039224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640094a0 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06ffff700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 0x7ff070159b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff064014040 tx=0x7ff064004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0640038a0 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06401e070 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07006bd00 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064003c60 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.941+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0700b3960 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.942+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0700751e0 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.943+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff06402d450 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.943+0000 7ff06cff9700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff05c06b350 0x7ff05c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.943+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff06408b040 con 0x7ff0700ae910 2024-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.943+0000 7ff06f7fe700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff05c06b350 0x7ff05c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.944+0000 7ff06f7fe700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff05c06b350 0x7ff05c06d810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff07015f2a0 tx=0x7ff060008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:40.947+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff064056020 con 0x7ff0700ae910 2024-01-15T14:35:41.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.139+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0700b5970 con 0x7ff0700ae910 2024-01-15T14:35:41.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.140+0000 7ff06cff9700 1 -- 172.21.15.77:0/3128039224 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7ff064059640 con 0x7ff0700ae910 2024-01-15T14:35:41.141 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:41.142 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff05c06b350 msgr2=0x7ff05c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff05c06b350 0x7ff05c06d810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff07015f2a0 tx=0x7ff060008040 comp rx=0 tx=0).stop 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 msgr2=0x7ff070159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 0x7ff070159b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff064014040 tx=0x7ff064004750 comp rx=0 tx=0).stop 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 shutdown_connections 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff05c06b350 0x7ff05c06d810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff0700c4790 0x7ff0701f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0700ae910 0x7ff070159b90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 --2- 172.21.15.77:0/3128039224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff0700cc8c0 0x7ff0700c4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 >> 172.21.15.77:0/3128039224 conn(0x7ff0700730a0 msgr2=0x7ff0700cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:41.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 shutdown_connections 2024-01-15T14:35:41.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.142+0000 7ff07632a700 1 -- 172.21.15.77:0/3128039224 wait complete. 2024-01-15T14:35:41.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:41.156 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.281+0000 7f53e0e79700 1 Processor -- start 2024-01-15T14:35:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.281+0000 7f53e0e79700 1 -- start start 2024-01-15T14:35:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53e0e79700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc15e280 0x7f53dc0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53e0e79700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53dc073f90 0x7f53dc0b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53e0e79700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f53dc0b5050 0x7f53dc0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53e0e79700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53dc0b5500 con 0x7f53dc073f90 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53e0e79700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53dc0b42a0 con 0x7f53dc0b5050 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53e0e79700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc0af900 con 0x7f53dc15e280 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53dc073f90 0x7f53dc0b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53dc073f90 0x7f53dc0b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40414/0 (socket says 172.21.15.77:40414) 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 -- 172.21.15.77:0/1592501020 learned_addr learned my addr 172.21.15.77:0/1592501020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53dad9d700 1 --2- 172.21.15.77:0/1592501020 >> v2:172.21.15.159:3300/0 conn(0x7f53dc0b5050 0x7f53dc0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53da59c700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc15e280 0x7f53dc0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 -- 172.21.15.77:0/1592501020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc15e280 msgr2=0x7f53dc0b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc15e280 0x7f53dc0b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 -- 172.21.15.77:0/1592501020 >> v2:172.21.15.159:3300/0 conn(0x7f53dc0b5050 msgr2=0x7f53dc0ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 --2- 172.21.15.77:0/1592501020 >> v2:172.21.15.159:3300/0 conn(0x7f53dc0b5050 0x7f53dc0ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 -- 172.21.15.77:0/1592501020 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc074d30 con 0x7f53dc073f90 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53da59c700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc15e280 0x7f53dc0b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d9d9b700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc073f90 0x7f53dc0b4b10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0731f0 tx=0x7f53d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fca37940ae72bd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d959a700 1 -- 172.21.15.77:0/1592501020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0011830 con 0x7f53dc073f90 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d959a700 1 -- 172.21.15.77:0/1592501020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d0011990 con 0x7f53dc073f90 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.282+0000 7f53d959a700 1 -- 172.21.15.77:0/1592501020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0011d20 con 0x7f53dc073f90 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 -- 172.21.15.77:0/1592501020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc073f90 msgr2=0x7f53dc0b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc073f90 0x7f53dc0b4b10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0731f0 tx=0x7f53d0009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 -- 172.21.15.77:0/1592501020 shutdown_connections 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 --2- 172.21.15.77:0/1592501020 >> v2:172.21.15.159:3300/0 conn(0x7f53dc0b5050 0x7f53dc0ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc15e280 0x7f53dc0b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 --2- 172.21.15.77:0/1592501020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc073f90 0x7f53dc0b4b10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 -- 172.21.15.77:0/1592501020 >> 172.21.15.77:0/1592501020 conn(0x7f53dc072b20 msgr2=0x7f53dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 -- 172.21.15.77:0/1592501020 shutdown_connections 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.283+0000 7f53e0e79700 1 -- 172.21.15.77:0/1592501020 wait complete. 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.284+0000 7f53e0e79700 1 Processor -- start 2024-01-15T14:35:41.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.284+0000 7f53e0e79700 1 -- start start 2024-01-15T14:35:41.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53e0e79700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc073f90 0x7f53dc15f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53e0e79700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53dc0b5050 0x7f53dc161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53e0e79700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 0x7f53dc1f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53e0e79700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc15ed10 con 0x7f53dc1f3840 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53e0e79700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc15f670 con 0x7f53dc0b5050 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53e0e79700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc15e6e0 con 0x7f53dc073f90 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53dad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 0x7f53dc1f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53dad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 0x7f53dc1f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40420/0 (socket says 172.21.15.77:40420) 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53dad9d700 1 -- 172.21.15.77:0/2102255978 learned_addr learned my addr 172.21.15.77:0/2102255978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53da59c700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc073f90 0x7f53dc15f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.285+0000 7f53d9d9b700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53dc0b5050 0x7f53dc161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53dad9d700 1 -- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc073f90 msgr2=0x7f53dc15f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53dad9d700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc073f90 0x7f53dc15f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53dad9d700 1 -- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53dc0b5050 msgr2=0x7f53dc161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53dad9d700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53dc0b5050 0x7f53dc161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53dad9d700 1 -- 172.21.15.77:0/2102255978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0009550 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53dad9d700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 0x7f53dc1f3cf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53d400ef30 tx=0x7f53d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4010040 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d400ba20 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53dc1f4340 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d400bdd0 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.286+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53dc15e930 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.287+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53dc1f4860 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.287+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f53d4014400 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.288+0000 7f53caffd700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f53bc06b350 0x7f53bc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.288+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53d4089dd0 con 0x7f53dc1f3840 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.288+0000 7f53da59c700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f53bc06b350 0x7f53bc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.288+0000 7f53da59c700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f53bc06b350 0x7f53bc06d810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53cc006f10 tx=0x7f53cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.292+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53d4054cd0 con 0x7f53dc1f3840 2024-01-15T14:35:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.482+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53dc15ee50 con 0x7f53dc1f3840 2024-01-15T14:35:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.482+0000 7f53caffd700 1 -- 172.21.15.77:0/2102255978 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53d408b020 con 0x7f53dc1f3840 2024-01-15T14:35:41.484 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f53bc06b350 msgr2=0x7f53bc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f53bc06b350 0x7f53bc06d810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53cc006f10 tx=0x7f53cc008040 comp rx=0 tx=0).stop 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 msgr2=0x7f53dc1f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 0x7f53dc1f3cf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53d400ef30 tx=0x7f53d400ef60 comp rx=0 tx=0).stop 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 shutdown_connections 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f53bc06b350 0x7f53bc06d810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f53dc073f90 0x7f53dc15f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53dc1f3840 0x7f53dc1f3cf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 --2- 172.21.15.77:0/2102255978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f53dc0b5050 0x7f53dc161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.484+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 >> 172.21.15.77:0/2102255978 conn(0x7f53dc072b20 msgr2=0x7f53dc06c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:41.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.485+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 shutdown_connections 2024-01-15T14:35:41.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.485+0000 7f53e0e79700 1 -- 172.21.15.77:0/2102255978 wait complete. 2024-01-15T14:35:41.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:41.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.621+0000 7f8c22d16700 1 Processor -- start 2024-01-15T14:35:41.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.621+0000 7f8c22d16700 1 -- start start 2024-01-15T14:35:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c22d16700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c22d16700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1c0d0a70 con 0x7f8c1c09b110 2024-01-15T14:35:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c22d16700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1c0d1c10 con 0x7f8c1c161a70 2024-01-15T14:35:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c22d16700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0b2ab0 con 0x7f8c1c161690 2024-01-15T14:35:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c212b3700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8c1c09b110 0x7f8c1c0cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47238/0 (socket says 172.21.15.77:47238) 2024-01-15T14:35:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c212b3700 1 -- 172.21.15.77:0/4184725267 learned_addr learned my addr 172.21.15.77:0/4184725267 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c20ab2700 1 --2- 172.21.15.77:0/4184725267 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327913453 0 0) 0x7f8c1c0d0a70 con 0x7f8c1c09b110 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.622+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c0035b0 con 0x7f8c1c09b110 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17893106 0 0) 0x7f8c1c0d1c10 con 0x7f8c1c161a70 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1c0d0a70 con 0x7f8c1c161a70 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916481520 0 0) 0x7f8c0c0035b0 con 0x7f8c1c09b110 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c20ab2700 1 -- 172.21.15.77:0/4184725267 >> v1:172.21.15.159:6789/0 conn(0x7f8c1c161a70 legacy=0x7f8c1c09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c20ab2700 1 -- 172.21.15.77:0/4184725267 >> v1:172.21.15.77:6789/0 conn(0x7f8c1c09b110 legacy=0x7f8c1c0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c20ab2700 1 -- 172.21.15.77:0/4184725267 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c0b2d90 con 0x7f8c1c161690 2024-01-15T14:35:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c20ab2700 1 --2- 172.21.15.77:0/4184725267 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c0d0f20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009870 tx=0x7f8c10009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd13ad54397ab19 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c140019f0 con 0x7f8c1c161690 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c10004030 con 0x7f8c1c161690 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.623+0000 7f8c1b7fe700 1 -- 172.21.15.77:0/4184725267 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c100043a0 con 0x7f8c1c161690 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 -- 172.21.15.77:0/4184725267 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 msgr2=0x7f8c1c0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 --2- 172.21.15.77:0/4184725267 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c0d0f20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009870 tx=0x7f8c10009b80 comp rx=0 tx=0).stop 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 -- 172.21.15.77:0/4184725267 shutdown_connections 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 --2- 172.21.15.77:0/4184725267 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c0d0f20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 -- 172.21.15.77:0/4184725267 >> 172.21.15.77:0/4184725267 conn(0x7f8c1c0d1e50 msgr2=0x7f8c1c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 -- 172.21.15.77:0/4184725267 shutdown_connections 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.624+0000 7f8c22d16700 1 -- 172.21.15.77:0/4184725267 wait complete. 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.625+0000 7f8c22d16700 1 Processor -- start 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.625+0000 7f8c22d16700 1 -- start start 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c1c09b110 0x7f8c1c15a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c15f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 0x7f8c1c15f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0caae0 con 0x7f8c1c161a70 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c15fe20 con 0x7f8c1c09b110 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0c9e60 con 0x7f8c1c161690 2024-01-15T14:35:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 0x7f8c1c15f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 0x7f8c1c15f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40424/0 (socket says 172.21.15.77:40424) 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 -- 172.21.15.77:0/2800783471 learned_addr learned my addr 172.21.15.77:0/2800783471 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c20ab2700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c1c09b110 0x7f8c1c15a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c1bfff700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c15f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 -- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 msgr2=0x7f8c1c15f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c15f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 -- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c1c09b110 msgr2=0x7f8c1c15a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c1c09b110 0x7f8c1c15a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 -- 172.21.15.77:0/2800783471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c100094a0 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c212b3700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 0x7f8c1c15f8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8c14001700 tx=0x7f8c1400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1400ac20 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c0ca140 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1400ad80 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.626+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14007310 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.627+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c0ca700 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.627+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1c069020 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.628+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8c14007470 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.628+0000 7f8c197fa700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c0006b350 0x7f8c0006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.628+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c14089700 con 0x7f8c1c161a70 2024-01-15T14:35:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.628+0000 7f8c20ab2700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c0006b350 0x7f8c0006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.629+0000 7f8c20ab2700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c0006b350 0x7f8c0006d810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c100097c0 tx=0x7f8c10008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.633+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c14054580 con 0x7f8c1c161a70 2024-01-15T14:35:41.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.824+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c1c0cc920 con 0x7f8c1c161a70 2024-01-15T14:35:41.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.824+0000 7f8c197fa700 1 -- 172.21.15.77:0/2800783471 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c14057ba0 con 0x7f8c1c161a70 2024-01-15T14:35:41.826 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:41.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c0006b350 msgr2=0x7f8c0006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c0006b350 0x7f8c0006d810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c100097c0 tx=0x7f8c10008030 comp rx=0 tx=0).stop 2024-01-15T14:35:41.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 msgr2=0x7f8c1c15f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 0x7f8c1c15f8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8c14001700 tx=0x7f8c1400a060 comp rx=0 tx=0).stop 2024-01-15T14:35:41.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 shutdown_connections 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8c0006b350 0x7f8c0006d810 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c161a70 0x7f8c1c15f8e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8c1c09b110 0x7f8c1c15a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 --2- 172.21.15.77:0/2800783471 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8c1c161690 0x7f8c1c15f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 >> 172.21.15.77:0/2800783471 conn(0x7f8c1c0d1e50 msgr2=0x7f8c1c160510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 shutdown_connections 2024-01-15T14:35:41.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.826+0000 7f8c22d16700 1 -- 172.21.15.77:0/2800783471 wait complete. 2024-01-15T14:35:41.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.963+0000 7fb901a0e700 1 Processor -- start 2024-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.963+0000 7fb901a0e700 1 -- start start 2024-01-15T14:35:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb901a0e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 0x7fb8fc0d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb901a0e700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb8fc0b98e0 0x7fb8fc0d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb901a0e700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc162af0 con 0x7fb8fc161510 2024-01-15T14:35:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb901a0e700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0d13e0 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb901a0e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc161f70 con 0x7fb8fc0cf330 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb8fa7fc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 0x7fb8fc0d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb8fa7fc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 0x7fb8fc0d2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:46704/0 (socket says 172.21.15.77:46704) 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb8fa7fc700 1 -- 172.21.15.77:0/2662334807 learned_addr learned my addr 172.21.15.77:0/2662334807 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.964+0000 7fb8fb7fe700 1 --2- 172.21.15.77:0/2662334807 >> v2:172.21.15.159:3300/0 conn(0x7fb8fc0b98e0 0x7fb8fc0d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8fa7fc700 1 -- 172.21.15.77:0/2662334807 >> v2:172.21.15.159:3300/0 conn(0x7fb8fc0b98e0 msgr2=0x7fb8fc0d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8f9ffb700 1 -- 172.21.15.77:0/2662334807 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652186681 0 0) 0x7fb8fc162af0 con 0x7fb8fc161510 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8fa7fc700 1 --2- 172.21.15.77:0/2662334807 >> v2:172.21.15.159:3300/0 conn(0x7fb8fc0b98e0 0x7fb8fc0d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8fa7fc700 1 -- 172.21.15.77:0/2662334807 >> v1:172.21.15.159:6789/0 conn(0x7fb8fc161510 legacy=0x7fb8fc0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8fa7fc700 1 -- 172.21.15.77:0/2662334807 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc162250 con 0x7fb8fc0cf330 2024-01-15T14:35:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8fa7fc700 1 --2- 172.21.15.77:0/2662334807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 0x7fb8fc0d2700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0098f0 tx=0x7fb8ec009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b20be33f0cbbb108 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8f9ffb700 1 -- 172.21.15.77:0/2662334807 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec011830 con 0x7fb8fc0cf330 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8f9ffb700 1 -- 172.21.15.77:0/2662334807 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec011990 con 0x7fb8fc0cf330 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.965+0000 7fb8f9ffb700 1 -- 172.21.15.77:0/2662334807 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec011d20 con 0x7fb8fc0cf330 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 -- 172.21.15.77:0/2662334807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 msgr2=0x7fb8fc0d2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2662334807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 0x7fb8fc0d2700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0098f0 tx=0x7fb8ec009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 -- 172.21.15.77:0/2662334807 shutdown_connections 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2662334807 >> v2:172.21.15.159:3300/0 conn(0x7fb8fc0b98e0 0x7fb8fc0d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2662334807 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc0cf330 0x7fb8fc0d2700 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 -- 172.21.15.77:0/2662334807 >> 172.21.15.77:0/2662334807 conn(0x7fb8fc06b410 msgr2=0x7fb8fc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 -- 172.21.15.77:0/2662334807 shutdown_connections 2024-01-15T14:35:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.966+0000 7fb901a0e700 1 -- 172.21.15.77:0/2662334807 wait complete. 2024-01-15T14:35:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.967+0000 7fb901a0e700 1 Processor -- start 2024-01-15T14:35:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.967+0000 7fb901a0e700 1 -- start start 2024-01-15T14:35:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 0x7fb8fc0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8fc0cf330 0x7fb8fc0b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc161510 0x7fb8fc14bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc14da60 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0b3f70 con 0x7fb8fc0cf330 2024-01-15T14:35:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0b4250 con 0x7fb8fc161510 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 0x7fb8fc0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 0x7fb8fc0c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40432/0 (socket says 172.21.15.77:40432) 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 -- 172.21.15.77:0/2936620833 learned_addr learned my addr 172.21.15.77:0/2936620833 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8fb7fe700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc161510 0x7fb8fc14bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8fa7fc700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8fc0cf330 0x7fb8fc0b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 -- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc161510 msgr2=0x7fb8fc14bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc161510 0x7fb8fc14bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 -- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8fc0cf330 msgr2=0x7fb8fc0b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8fc0cf330 0x7fb8fc0b8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 -- 172.21.15.77:0/2936620833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec009550 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8faffd700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 0x7fb8fc0c95b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000c550 tx=0x7fb8f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f00054e0 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f0003680 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc0b4530 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0003a30 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.968+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc0c9af0 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.969+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc068450 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.970+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb8f0003b90 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.970+0000 7fb8e3fff700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb8e806b350 0x7fb8e806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.970+0000 7fb8fa7fc700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb8e806b350 0x7fb8e806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.970+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8f00890a0 con 0x7fb8fc0b98e0 2024-01-15T14:35:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.971+0000 7fb8fa7fc700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb8e806b350 0x7fb8e806d810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc156cd0 tx=0x7fb8ec008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:41.975+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8f0053e90 con 0x7fb8fc0b98e0 2024-01-15T14:35:42.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.164+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb8fc069020 con 0x7fb8fc0b98e0 2024-01-15T14:35:42.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.164+0000 7fb8e3fff700 1 -- 172.21.15.77:0/2936620833 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb8f00170a0 con 0x7fb8fc0b98e0 2024-01-15T14:35:42.166 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:42.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb8e806b350 msgr2=0x7fb8e806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb8e806b350 0x7fb8e806d810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc156cd0 tx=0x7fb8ec008030 comp rx=0 tx=0).stop 2024-01-15T14:35:42.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 msgr2=0x7fb8fc0c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 0x7fb8fc0c95b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000c550 tx=0x7fb8f000c910 comp rx=0 tx=0).stop 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 shutdown_connections 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb8e806b350 0x7fb8e806d810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb8fc161510 0x7fb8fc14bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8fc0b98e0 0x7fb8fc0c95b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 --2- 172.21.15.77:0/2936620833 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb8fc0cf330 0x7fb8fc0b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 >> 172.21.15.77:0/2936620833 conn(0x7fb8fc06b410 msgr2=0x7fb8fc1603c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 shutdown_connections 2024-01-15T14:35:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.167+0000 7fb901a0e700 1 -- 172.21.15.77:0/2936620833 wait complete. 2024-01-15T14:35:42.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T14:35:42.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.302+0000 7f1f446f5700 1 Processor -- start 2024-01-15T14:35:42.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.302+0000 7f1f446f5700 1 -- start start 2024-01-15T14:35:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f446f5700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1f3c15e080 0x7f1f3c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f446f5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f3c0d22f0 0x7f1f3c0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f446f5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c0d0150 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f446f5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c15e460 con 0x7f1f3c0d22f0 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f446f5700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c0cee50 con 0x7f1f3c15e080 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42c92700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f3c0d22f0 0x7f1f3c0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f41c90700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f3c0ce4d0 0x7f1f3c0d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47252/0 (socket says 172.21.15.77:47252) 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f41c90700 1 -- 172.21.15.77:0/2167987689 learned_addr learned my addr 172.21.15.77:0/2167987689 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42491700 1 --2- 172.21.15.77:0/2167987689 >> v2:172.21.15.159:3300/0 conn(0x7f1f3c15e080 0x7f1f3c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42c92700 1 -- 172.21.15.77:0/2167987689 >> v2:172.21.15.159:3300/0 conn(0x7f1f3c15e080 msgr2=0x7f1f3c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42c92700 1 --2- 172.21.15.77:0/2167987689 >> v2:172.21.15.159:3300/0 conn(0x7f1f3c15e080 0x7f1f3c0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42c92700 1 -- 172.21.15.77:0/2167987689 >> v1:172.21.15.77:6789/0 conn(0x7f1f3c0ce4d0 legacy=0x7f1f3c0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42c92700 1 -- 172.21.15.77:0/2167987689 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0cf130 con 0x7f1f3c0d22f0 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f4148f700 1 -- 172.21.15.77:0/2167987689 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793435508 0 0) 0x7f1f3c0d0150 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.303+0000 7f1f42c92700 1 --2- 172.21.15.77:0/2167987689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0d22f0 0x7f1f3c0d05e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c073070 tx=0x7f1f3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4785f0e4b6fddf83 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f4148f700 1 -- 172.21.15.77:0/2167987689 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400e070 con 0x7f1f3c0d22f0 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f4148f700 1 -- 172.21.15.77:0/2167987689 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3400bbd0 con 0x7f1f3c0d22f0 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f4148f700 1 -- 172.21.15.77:0/2167987689 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34012430 con 0x7f1f3c0d22f0 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 -- 172.21.15.77:0/2167987689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0d22f0 msgr2=0x7f1f3c0d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2167987689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0d22f0 0x7f1f3c0d05e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c073070 tx=0x7f1f3400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 -- 172.21.15.77:0/2167987689 shutdown_connections 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2167987689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0d22f0 0x7f1f3c0d05e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2167987689 >> v2:172.21.15.159:3300/0 conn(0x7f1f3c15e080 0x7f1f3c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 -- 172.21.15.77:0/2167987689 >> 172.21.15.77:0/2167987689 conn(0x7f1f3c073130 msgr2=0x7f1f3c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 -- 172.21.15.77:0/2167987689 shutdown_connections 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.304+0000 7f1f446f5700 1 -- 172.21.15.77:0/2167987689 wait complete. 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.305+0000 7f1f446f5700 1 Processor -- start 2024-01-15T14:35:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.305+0000 7f1f446f5700 1 -- start start 2024-01-15T14:35:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f446f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 0x7f1f3c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f446f5700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f3c0d22f0 0x7f1f3c0b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f446f5700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f3c15e080 0x7f1f3c0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f446f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0b5700 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f446f5700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c15c7b0 con 0x7f1f3c15e080 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f446f5700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0cb840 con 0x7f1f3c0d22f0 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 0x7f1f3c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 0x7f1f3c15c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40452/0 (socket says 172.21.15.77:40452) 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 -- 172.21.15.77:0/2136519734 learned_addr learned my addr 172.21.15.77:0/2136519734 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f41c90700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f3c0d22f0 0x7f1f3c0b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42c92700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f3c15e080 0x7f1f3c0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 -- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f3c0d22f0 msgr2=0x7f1f3c0b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f3c0d22f0 0x7f1f3c0b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 -- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f3c15e080 msgr2=0x7f1f3c0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.306+0000 7f1f42491700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f3c15e080 0x7f1f3c0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f42491700 1 -- 172.21.15.77:0/2136519734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f34009a70 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f42491700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 0x7f1f3c15c270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00e810 tx=0x7f1f2c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c013610 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c013770 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c0b5320 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c013b20 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.307+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c15d030 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.308+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c067f60 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.308+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1f2c013c80 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.309+0000 7f1f32ffd700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1f2006b350 0x7f1f2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.309+0000 7f1f41c90700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1f2006b350 0x7f1f2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.309+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f2c089840 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.309+0000 7f1f41c90700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1f2006b350 0x7f1f2006d810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c09bc20 tx=0x7f1f38007c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:42.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.313+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f2c0546c0 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.520+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1f3c0b7bd0 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.520+0000 7f1f32ffd700 1 -- 172.21.15.77:0/2136519734 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1f2c057ce0 con 0x7f1f3c0ce4d0 2024-01-15T14:35:42.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.522+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1f2006b350 msgr2=0x7f1f2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.522+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1f2006b350 0x7f1f2006d810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c09bc20 tx=0x7f1f38007c90 comp rx=0 tx=0).stop 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.522+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 msgr2=0x7f1f3c15c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.522+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 0x7f1f3c15c270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00e810 tx=0x7f1f2c00ebd0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 shutdown_connections 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1f2006b350 0x7f1f2006d810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1f3c15e080 0x7f1f3c0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f3c0ce4d0 0x7f1f3c15c270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 --2- 172.21.15.77:0/2136519734 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1f3c0d22f0 0x7f1f3c0b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 >> 172.21.15.77:0/2136519734 conn(0x7f1f3c073130 msgr2=0x7f1f3c0ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 shutdown_connections 2024-01-15T14:35:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.523+0000 7f1f446f5700 1 -- 172.21.15.77:0/2136519734 wait complete. 2024-01-15T14:35:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-01-15T14:35:42.536 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-01-15T14:35:42.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:42.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-01-15T14:35:42.576 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-01-15T14:35:42.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:42.576 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T14:35:42.633 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:42.633 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:42.655 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:42.655 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:42.717 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:35:42.717 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:35:42.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:42.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.899+0000 7fac43f3f700 1 Processor -- start 2024-01-15T14:35:42.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.899+0000 7fac43f3f700 1 -- start start 2024-01-15T14:35:42.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac43f3f700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fac3c152cc0 0x7fac3c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac43f3f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c15c4b0 con 0x7fac3c0b49a0 2024-01-15T14:35:42.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac43f3f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c15c790 con 0x7fac3c0d0630 2024-01-15T14:35:42.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac43f3f700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3c15ca70 con 0x7fac3c152cc0 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac414da700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fac3c0b49a0 0x7fac3c0cff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47258/0 (socket says 172.21.15.77:47258) 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac414da700 1 -- 172.21.15.77:0/1186459094 learned_addr learned my addr 172.21.15.77:0/1186459094 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac41cdb700 1 --2- 172.21.15.77:0/1186459094 >> v2:172.21.15.159:3300/0 conn(0x7fac3c152cc0 0x7fac3c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675399521 0 0) 0x7fac3c15c4b0 con 0x7fac3c0b49a0 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac280035b0 con 0x7fac3c0b49a0 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301064755 0 0) 0x7fac280035b0 con 0x7fac3c0b49a0 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 >> v2:172.21.15.159:3300/0 conn(0x7fac3c152cc0 msgr2=0x7fac3c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac40cd9700 1 --2- 172.21.15.77:0/1186459094 >> v2:172.21.15.159:3300/0 conn(0x7fac3c152cc0 0x7fac3c06c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.900+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 >> v1:172.21.15.159:6789/0 conn(0x7fac3c0d0630 legacy=0x7fac3c15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c15cd50 con 0x7fac3c0b49a0 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fac38001670 con 0x7fac3c0b49a0 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 >> v1:172.21.15.77:6789/0 conn(0x7fac3c0b49a0 legacy=0x7fac3c0cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac28004850 0x7fac28004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac28005210 0x7fac2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c15ca70 con 0x7fac28003a00 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2800c1b0 con 0x7fac28004850 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2800c370 con 0x7fac28005210 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac414da700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac28004850 0x7fac28004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac424dc700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac28005210 0x7fac2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 -- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac28005210 msgr2=0x7fac2800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac28005210 0x7fac2800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 -- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac28004850 msgr2=0x7fac28004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac28004850 0x7fac28004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 -- 172.21.15.77:0/1186459094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac280035b0 con 0x7fac28003a00 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac424dc700 1 -- 172.21.15.77:0/1186459094 reap_dead start 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac41cdb700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac28003e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009b60 tx=0x7fac28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbeb4cf63aa3a89 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00e6c0 con 0x7fac28003a00 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c00e820 con 0x7fac28003a00 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.901+0000 7fac40cd9700 1 -- 172.21.15.77:0/1186459094 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00ebb0 con 0x7fac28003a00 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 -- 172.21.15.77:0/1186459094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 msgr2=0x7fac28003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac28003e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009b60 tx=0x7fac28005880 comp rx=0 tx=0).stop 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 -- 172.21.15.77:0/1186459094 shutdown_connections 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 --2- 172.21.15.77:0/1186459094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac28003e20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 -- 172.21.15.77:0/1186459094 >> 172.21.15.77:0/1186459094 conn(0x7fac3c071b90 msgr2=0x7fac3c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 -- 172.21.15.77:0/1186459094 shutdown_connections 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.902+0000 7fac43f3f700 1 -- 172.21.15.77:0/1186459094 wait complete. 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.903+0000 7fac43f3f700 1 Processor -- start 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 -- start start 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac3c13d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac3c147460 0x7fac3c149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac3c14a230 0x7fac3c148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c148740 con 0x7fac28003a00 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c13d990 con 0x7fac3c14a230 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac43f3f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c14a6d0 con 0x7fac3c147460 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac41cdb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac3c13d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.904+0000 7fac41cdb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac3c13d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40466/0 (socket says 172.21.15.77:40466) 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 -- 172.21.15.77:0/3659011405 learned_addr learned my addr 172.21.15.77:0/3659011405 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac424dc700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac3c14a230 0x7fac3c148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac414da700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac3c147460 0x7fac3c149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 -- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac3c147460 msgr2=0x7fac3c149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac3c147460 0x7fac3c149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 -- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac3c14a230 msgr2=0x7fac3c148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac3c14a230 0x7fac3c148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 -- 172.21.15.77:0/3659011405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c14a9b0 con 0x7fac28003a00 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac424dc700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac3c14a230 0x7fac3c148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac414da700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac3c147460 0x7fac3c149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac41cdb700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac3c13d450 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00e4b0 tx=0x7fac2c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00e6c0 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c003ab0 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c003e60 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c1480f0 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.905+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c149110 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.906+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3c148a20 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.906+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac2c013470 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.907+0000 7fac327fc700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fac2006b350 0x7fac2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.907+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac2c08b220 con 0x7fac28003a00 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.907+0000 7fac414da700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fac2006b350 0x7fac2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.907+0000 7fac414da700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fac2006b350 0x7fac2006d810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fac3c147860 tx=0x7fac38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:42.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:42.912+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac2c056120 con 0x7fac28003a00 2024-01-15T14:35:43.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.102+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac3c0753f0 con 0x7fac28003a00 2024-01-15T14:35:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.103+0000 7fac327fc700 1 -- 172.21.15.77:0/3659011405 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fac2c059740 con 0x7fac28003a00 2024-01-15T14:35:43.105 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:43.105 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:06.213908+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fac2006b350 msgr2=0x7fac2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fac2006b350 0x7fac2006d810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fac3c147860 tx=0x7fac38009040 comp rx=0 tx=0).stop 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 msgr2=0x7fac3c13d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac3c13d450 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00e4b0 tx=0x7fac2c0047c0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 shutdown_connections 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fac2006b350 0x7fac2006d810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fac3c14a230 0x7fac3c148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac28003a00 0x7fac3c13d450 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 --2- 172.21.15.77:0/3659011405 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fac3c147460 0x7fac3c149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 >> 172.21.15.77:0/3659011405 conn(0x7fac3c071b90 msgr2=0x7fac3c0b4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 shutdown_connections 2024-01-15T14:35:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.105+0000 7fac43f3f700 1 -- 172.21.15.77:0/3659011405 wait complete. 2024-01-15T14:35:43.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:43.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:43.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.244+0000 7f2e4ae40700 1 Processor -- start 2024-01-15T14:35:43.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.244+0000 7f2e4ae40700 1 -- start start 2024-01-15T14:35:43.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e4ae40700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 0x7f2e44166af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e4ae40700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e440d0790 con 0x7f2e4415bee0 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e4ae40700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4406ee70 con 0x7f2e440d01b0 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e4ae40700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4406f150 con 0x7f2e440ceb20 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e48bdc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2e4415bee0 0x7f2e440cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47266/0 (socket says 172.21.15.77:47266) 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e48bdc700 1 -- 172.21.15.77:0/2756092841 learned_addr learned my addr 172.21.15.77:0/2756092841 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943994451 0 0) 0x7f2e440d0790 con 0x7f2e4415bee0 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e340035b0 con 0x7f2e4415bee0 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e493dd700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 0x7f2e44166af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2576659456 0 0) 0x7f2e340035b0 con 0x7f2e4415bee0 2024-01-15T14:35:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 msgr2=0x7f2e44166af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 0x7f2e44166af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 >> v1:172.21.15.159:6789/0 conn(0x7f2e440d01b0 legacy=0x7f2e440ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4406f430 con 0x7f2e4415bee0 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f2e38001670 con 0x7f2e4415bee0 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 >> v1:172.21.15.77:6789/0 conn(0x7f2e4415bee0 legacy=0x7f2e440cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e34003a30 0x7f2e34003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 0x7f2e34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4406f150 con 0x7f2e34004d90 2024-01-15T14:35:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3400ff80 con 0x7f2e34003a30 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.245+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34010140 con 0x7f2e440ceb20 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e48bdc700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e34003a30 0x7f2e34003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e43fff700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 0x7f2e34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 -- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 msgr2=0x7f2e34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e440ceb20 0x7f2e34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 -- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e34003a30 msgr2=0x7f2e34003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e34003a30 0x7f2e34003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 -- 172.21.15.77:0/2756092841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340035b0 con 0x7f2e34004d90 2024-01-15T14:35:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e493dd700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e34005210 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009f90 tx=0x7f2e34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3200257c873b8a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c00ec30 con 0x7f2e34004d90 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c00f040 con 0x7f2e34004d90 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.246+0000 7f2e437fe700 1 -- 172.21.15.77:0/2756092841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c014480 con 0x7f2e34004d90 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e4ae40700 1 -- 172.21.15.77:0/2756092841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 msgr2=0x7f2e34005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/2756092841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e34005210 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009f90 tx=0x7f2e34005880 comp rx=0 tx=0).stop 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e493dd700 1 -- 172.21.15.77:0/2756092841 reap_dead start 2024-01-15T14:35:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e4ae40700 1 -- 172.21.15.77:0/2756092841 shutdown_connections 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e4ae40700 1 -- 172.21.15.77:0/2756092841 >> 172.21.15.77:0/2756092841 conn(0x7f2e440b3780 msgr2=0x7f2e440b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e4ae40700 1 -- 172.21.15.77:0/2756092841 shutdown_connections 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.247+0000 7f2e4ae40700 1 -- 172.21.15.77:0/2756092841 wait complete. 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.248+0000 7f2e4ae40700 1 Processor -- start 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.248+0000 7f2e4ae40700 1 -- start start 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e4ae40700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e44157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e4ae40700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e44157710 0x7f2e440c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e4ae40700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e440b3b90 0x7f2e440b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e4ae40700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440b4580 con 0x7f2e34004d90 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e4ae40700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4415d5b0 con 0x7f2e440b3b90 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e4ae40700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4415d800 con 0x7f2e44157710 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e48bdc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e44157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e48bdc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e44157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40480/0 (socket says 172.21.15.77:40480) 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.249+0000 7f2e48bdc700 1 -- 172.21.15.77:0/4049064279 learned_addr learned my addr 172.21.15.77:0/4049064279 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e43fff700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e44157710 0x7f2e440c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e493dd700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e440b3b90 0x7f2e440b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e48bdc700 1 -- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e44157710 msgr2=0x7f2e440c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e48bdc700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e44157710 0x7f2e440c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e48bdc700 1 -- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e440b3b90 msgr2=0x7f2e440b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e48bdc700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e440b3b90 0x7f2e440b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e48bdc700 1 -- 172.21.15.77:0/4049064279 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4415dae0 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e48bdc700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e44157c50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2e38007290 tx=0x7f2e38007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38001450 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3800f850 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800fc00 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4415ddc0 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.250+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e440c84e0 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.251+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e440c86a0 con 0x7f2e34004d90 2024-01-15T14:35:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.251+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2e3800fd60 con 0x7f2e34004d90 2024-01-15T14:35:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.252+0000 7f2e417fa700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2e2806b350 0x7f2e2806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.252+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e38089c20 con 0x7f2e34004d90 2024-01-15T14:35:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.252+0000 7f2e43fff700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2e2806b350 0x7f2e2806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.252+0000 7f2e43fff700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2e2806b350 0x7f2e2806d810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e4415d540 tx=0x7f2e30009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.256+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e38054aa0 con 0x7f2e34004d90 2024-01-15T14:35:43.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.444+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e4415e2e0 con 0x7f2e34004d90 2024-01-15T14:35:43.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.445+0000 7f2e417fa700 1 -- 172.21.15.77:0/4049064279 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e4415e2e0 con 0x7f2e34004d90 2024-01-15T14:35:43.446 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2e2806b350 msgr2=0x7f2e2806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2e2806b350 0x7f2e2806d810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e4415d540 tx=0x7f2e30009040 comp rx=0 tx=0).stop 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 msgr2=0x7f2e44157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e44157c50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2e38007290 tx=0x7f2e38007650 comp rx=0 tx=0).stop 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 shutdown_connections 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2e2806b350 0x7f2e2806d810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2e440b3b90 0x7f2e440b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e34004d90 0x7f2e44157c50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 --2- 172.21.15.77:0/4049064279 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2e44157710 0x7f2e440c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 >> 172.21.15.77:0/4049064279 conn(0x7f2e440b3780 msgr2=0x7f2e44159390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:43.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 shutdown_connections 2024-01-15T14:35:43.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.447+0000 7f2e4ae40700 1 -- 172.21.15.77:0/4049064279 wait complete. 2024-01-15T14:35:43.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.583+0000 7f4d4edc3700 1 Processor -- start 2024-01-15T14:35:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.583+0000 7f4d4edc3700 1 -- start start 2024-01-15T14:35:43.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d4edc3700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4d480afb80 0x7f4d48073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d4edc3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4815e4a0 con 0x7f4d4815e0c0 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d4edc3700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4806aff0 con 0x7f4d480744d0 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d4edc3700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d480b0010 con 0x7f4d480afb80 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d4cb5f700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4d4815e0c0 0x7f4d480b3bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47282/0 (socket says 172.21.15.77:47282) 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d4cb5f700 1 -- 172.21.15.77:0/783601431 learned_addr learned my addr 172.21.15.77:0/783601431 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d47fff700 1 --2- 172.21.15.77:0/783601431 >> v2:172.21.15.159:3300/0 conn(0x7f4d480afb80 0x7f4d48073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.584+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816117680 0 0) 0x7f4d4815e4a0 con 0x7f4d4815e0c0 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d380035b0 con 0x7f4d4815e0c0 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2595018690 0 0) 0x7f4d380035b0 con 0x7f4d4815e0c0 2024-01-15T14:35:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 >> v2:172.21.15.159:3300/0 conn(0x7f4d480afb80 msgr2=0x7f4d48073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 --2- 172.21.15.77:0/783601431 >> v2:172.21.15.159:3300/0 conn(0x7f4d480afb80 0x7f4d48073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 >> v1:172.21.15.159:6789/0 conn(0x7f4d480744d0 legacy=0x7f4d480add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4806b2d0 con 0x7f4d4815e0c0 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f4d3c001670 con 0x7f4d4815e0c0 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 >> v1:172.21.15.77:6789/0 conn(0x7f4d4815e0c0 legacy=0x7f4d480b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38003a30 0x7f4d38003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d38004850 0x7f4d38004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d38005210 0x7f4d3800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d480b0010 con 0x7f4d38005210 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3800c0e0 con 0x7f4d38003a30 2024-01-15T14:35:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3800c2a0 con 0x7f4d38004850 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d38005210 0x7f4d3800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d47fff700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d38004850 0x7f4d38004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4cb5f700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38003a30 0x7f4d38003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 -- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d38004850 msgr2=0x7f4d38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d38004850 0x7f4d38004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 -- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38003a30 msgr2=0x7f4d38003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38003a30 0x7f4d38003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 -- 172.21.15.77:0/783601431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380035b0 con 0x7f4d38005210 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4cb5f700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38003a30 0x7f4d38003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 -- 172.21.15.77:0/783601431 reap_dead start 2024-01-15T14:35:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d4d360700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d38005210 0x7f4d3800ba10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000c190 tx=0x7f4d38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df3a0abd3a5dcb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40001db0 con 0x7f4d38005210 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.585+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4000ccf0 con 0x7f4d38005210 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d477fe700 1 -- 172.21.15.77:0/783601431 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40003c40 con 0x7f4d38005210 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 -- 172.21.15.77:0/783601431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d38005210 msgr2=0x7f4d3800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d38005210 0x7f4d3800ba10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000c190 tx=0x7f4d38005880 comp rx=0 tx=0).stop 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 -- 172.21.15.77:0/783601431 shutdown_connections 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/783601431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d38005210 0x7f4d3800ba10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 -- 172.21.15.77:0/783601431 >> 172.21.15.77:0/783601431 conn(0x7f4d480d0c00 msgr2=0x7f4d480d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 -- 172.21.15.77:0/783601431 shutdown_connections 2024-01-15T14:35:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.586+0000 7f4d4edc3700 1 -- 172.21.15.77:0/783601431 wait complete. 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.587+0000 7f4d4edc3700 1 Processor -- start 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.587+0000 7f4d4edc3700 1 -- start start 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4edc3700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38005210 0x7f4d48160e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4edc3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 0x7f4d480cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4edc3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d481f3e90 0x7f4d480b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4edc3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48161a40 con 0x7f4d4815e690 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4edc3700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d481f3480 con 0x7f4d38005210 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4edc3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d481f3a00 con 0x7f4d481f3e90 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d47fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 0x7f4d480cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d47fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 0x7f4d480cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40496/0 (socket says 172.21.15.77:40496) 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d47fff700 1 -- 172.21.15.77:0/2024639420 learned_addr learned my addr 172.21.15.77:0/2024639420 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4cb5f700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38005210 0x7f4d48160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.588+0000 7f4d4d360700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d481f3e90 0x7f4d480b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d47fff700 1 -- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d481f3e90 msgr2=0x7f4d480b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d47fff700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d481f3e90 0x7f4d480b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d47fff700 1 -- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38005210 msgr2=0x7f4d48160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d47fff700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38005210 0x7f4d48160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d47fff700 1 -- 172.21.15.77:0/2024639420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48161480 con 0x7f4d4815e690 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d4cb5f700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38005210 0x7f4d48160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d47fff700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 0x7f4d480cf910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4d340119a0 tx=0x7f4d34011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400f940 con 0x7f4d4815e690 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3400faa0 con 0x7f4d4815e690 2024-01-15T14:35:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400fe30 con 0x7f4d4815e690 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d48161760 con 0x7f4d4815e690 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.589+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d481f4340 con 0x7f4d4815e690 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.590+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d481f3bb0 con 0x7f4d4815e690 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.590+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4d34014440 con 0x7f4d4815e690 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.591+0000 7f4d457fa700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4d2c06b350 0x7f4d2c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.591+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d3408a200 con 0x7f4d4815e690 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.591+0000 7f4d4cb5f700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4d2c06b350 0x7f4d2c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.591+0000 7f4d4cb5f700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4d2c06b350 0x7f4d2c06d810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4d480d0340 tx=0x7f4d3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.595+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d481f3bb0 con 0x7f4d4815e690 2024-01-15T14:35:43.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.783+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d481f3b70 con 0x7f4d4815e690 2024-01-15T14:35:43.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.783+0000 7f4d457fa700 1 -- 172.21.15.77:0/2024639420 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4d34058c40 con 0x7f4d4815e690 2024-01-15T14:35:43.785 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:43.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.785+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4d2c06b350 msgr2=0x7f4d2c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.785+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4d2c06b350 0x7f4d2c06d810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4d480d0340 tx=0x7f4d3c009040 comp rx=0 tx=0).stop 2024-01-15T14:35:43.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.785+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 msgr2=0x7f4d480cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.785+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 0x7f4d480cf910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4d340119a0 tx=0x7f4d34011d60 comp rx=0 tx=0).stop 2024-01-15T14:35:43.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 shutdown_connections 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4d2c06b350 0x7f4d2c06d810 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4d481f3e90 0x7f4d480b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4d38005210 0x7f4d48160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 --2- 172.21.15.77:0/2024639420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d4815e690 0x7f4d480cf910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 >> 172.21.15.77:0/2024639420 conn(0x7f4d480d0c00 msgr2=0x7f4d480add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 shutdown_connections 2024-01-15T14:35:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.786+0000 7f4d4edc3700 1 -- 172.21.15.77:0/2024639420 wait complete. 2024-01-15T14:35:43.799 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:43.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.921+0000 7ff7c5033700 1 Processor -- start 2024-01-15T14:35:43.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.921+0000 7ff7c5033700 1 -- start start 2024-01-15T14:35:43.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.922+0000 7ff7c5033700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7c00b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.922+0000 7ff7c5033700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c006d730 con 0x7ff7c01584d0 2024-01-15T14:35:43.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.922+0000 7ff7c5033700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c015f540 con 0x7ff7c00cfe70 2024-01-15T14:35:43.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.922+0000 7ff7c5033700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c006cb20 con 0x7ff7c00d0320 2024-01-15T14:35:43.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.922+0000 7ff7bed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff7c01584d0 0x7ff7c00b45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47294/0 (socket says 172.21.15.77:47294) 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.922+0000 7ff7bed9d700 1 -- 172.21.15.77:0/1278222134 learned_addr learned my addr 172.21.15.77:0/1278222134 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7c00b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631510066 0 0) 0x7ff7c006d730 con 0x7ff7c01584d0 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7ac0035b0 con 0x7ff7c01584d0 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239619287 0 0) 0x7ff7c015f540 con 0x7ff7c00cfe70 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c006d730 con 0x7ff7c00cfe70 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671174764 0 0) 0x7ff7ac0035b0 con 0x7ff7c01584d0 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 msgr2=0x7ff7c00b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7c00b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 >> v1:172.21.15.159:6789/0 conn(0x7ff7c00cfe70 legacy=0x7ff7c00b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c006ce00 con 0x7ff7c01584d0 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7ff7a8001670 con 0x7ff7c01584d0 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7c00b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 >> v1:172.21.15.77:6789/0 conn(0x7ff7c01584d0 legacy=0x7ff7c00b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7ac003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7ac004150 0x7ff7ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ac004d90 0x7ff7ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c006cb20 con 0x7ff7ac004d90 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac00bb20 con 0x7ff7ac004150 2024-01-15T14:35:43.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac00bce0 con 0x7ff7c00d0320 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ac004d90 0x7ff7ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bed9d700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7ac003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7be59c700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7ac004150 0x7ff7ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 msgr2=0x7ff7ac003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c00d0320 0x7ff7ac003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7ac004150 msgr2=0x7ff7ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7ac004150 0x7ff7ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.923+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1278222134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8009a70 con 0x7ff7ac004d90 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ac004d90 0x7ff7ac005210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800e320 tx=0x7ff7ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186b8551144d571d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b800ede0 con 0x7ff7ac004d90 2024-01-15T14:35:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b800f040 con 0x7ff7ac004d90 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7bdd9b700 1 -- 172.21.15.77:0/1278222134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8003c60 con 0x7ff7ac004d90 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7c5033700 1 -- 172.21.15.77:0/1278222134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ac004d90 msgr2=0x7ff7ac005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1278222134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ac004d90 0x7ff7ac005210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800e320 tx=0x7ff7ac005880 comp rx=0 tx=0).stop 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1278222134 reap_dead start 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7c5033700 1 -- 172.21.15.77:0/1278222134 shutdown_connections 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7c5033700 1 -- 172.21.15.77:0/1278222134 >> 172.21.15.77:0/1278222134 conn(0x7ff7c006b410 msgr2=0x7ff7c006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7c5033700 1 -- 172.21.15.77:0/1278222134 shutdown_connections 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.924+0000 7ff7c5033700 1 -- 172.21.15.77:0/1278222134 wait complete. 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.926+0000 7ff7c5033700 1 Processor -- start 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.926+0000 7ff7c5033700 1 -- start start 2024-01-15T14:35:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7c0146120 0x7ff7c0148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c0148ed0 0x7ff7c00b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 0x7ff7c00b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0146d80 con 0x7ff7c015d690 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0149910 con 0x7ff7c0146120 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0149350 con 0x7ff7c0148ed0 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 0x7ff7c00b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 0x7ff7c00b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40518/0 (socket says 172.21.15.77:40518) 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1855986373 learned_addr learned my addr 172.21.15.77:0/1855986373 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7be59c700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c0148ed0 0x7ff7c00b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bed9d700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7c0146120 0x7ff7c0148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c0148ed0 msgr2=0x7ff7c00b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c0148ed0 0x7ff7c00b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7c0146120 msgr2=0x7ff7c0148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7c0146120 0x7ff7c0148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 -- 172.21.15.77:0/1855986373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0149630 con 0x7ff7c015d690 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7bf59e700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 0x7ff7c00b3aa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00baf0 tx=0x7ff7b8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8004440 con 0x7ff7c015d690 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b80045a0 con 0x7ff7c015d690 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8004950 con 0x7ff7c015d690 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c0146a10 con 0x7ff7c015d690 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.927+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c01466e0 con 0x7ff7c015d690 2024-01-15T14:35:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.928+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c0146f30 con 0x7ff7c015d690 2024-01-15T14:35:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.929+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff7b8024020 con 0x7ff7c015d690 2024-01-15T14:35:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.929+0000 7ff7b77fe700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff7a006b350 0x7ff7a006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.929+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7b808bac0 con 0x7ff7c015d690 2024-01-15T14:35:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.929+0000 7ff7bed9d700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff7a006b350 0x7ff7a006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.930+0000 7ff7bed9d700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff7a006b350 0x7ff7a006d810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff7a80048d0 tx=0x7ff7a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:43.934+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7c0146f30 con 0x7ff7c015d690 2024-01-15T14:35:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.124+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff7c0069020 con 0x7ff7c015d690 2024-01-15T14:35:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.124+0000 7ff7b77fe700 1 -- 172.21.15.77:0/1855986373 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff7b808be60 con 0x7ff7c015d690 2024-01-15T14:35:44.126 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff7a006b350 msgr2=0x7ff7a006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff7a006b350 0x7ff7a006d810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff7a80048d0 tx=0x7ff7a8009040 comp rx=0 tx=0).stop 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 msgr2=0x7ff7c00b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 0x7ff7c00b3aa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00baf0 tx=0x7ff7b8009750 comp rx=0 tx=0).stop 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 shutdown_connections 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff7a006b350 0x7ff7a006d810 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7c015d690 0x7ff7c00b3aa0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff7c0146120 0x7ff7c0148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 --2- 172.21.15.77:0/1855986373 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff7c0148ed0 0x7ff7c00b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.126+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 >> 172.21.15.77:0/1855986373 conn(0x7ff7c006b410 msgr2=0x7ff7c00afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.127+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 shutdown_connections 2024-01-15T14:35:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.127+0000 7ff7c5033700 1 -- 172.21.15.77:0/1855986373 wait complete. 2024-01-15T14:35:44.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T14:35:44.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.265+0000 7fd735a45700 1 Processor -- start 2024-01-15T14:35:44.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.265+0000 7fd735a45700 1 -- start start 2024-01-15T14:35:44.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd735a45700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd7300d1b00 0x7fd7300d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd735a45700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd7300b1cb0 0x7fd7300b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd735a45700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7300ce4a0 con 0x7fd73015e110 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd735a45700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7300d2490 con 0x7fd7300d1b00 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd735a45700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7300cefe0 con 0x7fd7300b1cb0 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72e7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd7300d1b00 0x7fd7300d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72effd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd73015e110 0x7fd7300b1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47300/0 (socket says 172.21.15.77:47300) 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72effd700 1 -- 172.21.15.77:0/289956347 learned_addr learned my addr 172.21.15.77:0/289956347 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72f7fe700 1 --2- 172.21.15.77:0/289956347 >> v2:172.21.15.159:3300/0 conn(0x7fd7300b1cb0 0x7fd7300b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72e7fc700 1 -- 172.21.15.77:0/289956347 >> v2:172.21.15.159:3300/0 conn(0x7fd7300b1cb0 msgr2=0x7fd7300b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72e7fc700 1 --2- 172.21.15.77:0/289956347 >> v2:172.21.15.159:3300/0 conn(0x7fd7300b1cb0 0x7fd7300b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72e7fc700 1 -- 172.21.15.77:0/289956347 >> v1:172.21.15.77:6789/0 conn(0x7fd73015e110 legacy=0x7fd7300b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72e7fc700 1 -- 172.21.15.77:0/289956347 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7300cf470 con 0x7fd7300d1b00 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72dffb700 1 -- 172.21.15.77:0/289956347 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102659682 0 0) 0x7fd7300ce4a0 con 0x7fd73015e110 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72f7fe700 1 --2- 172.21.15.77:0/289956347 >> v2:172.21.15.159:3300/0 conn(0x7fd7300b1cb0 0x7fd7300b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72e7fc700 1 --2- 172.21.15.77:0/289956347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300d1b00 0x7fd7300d1f50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd7300cc1f0 tx=0x7fd718009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e612faee37ea5fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.266+0000 7fd72dffb700 1 -- 172.21.15.77:0/289956347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718011830 con 0x7fd7300d1b00 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd72dffb700 1 -- 172.21.15.77:0/289956347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd718011990 con 0x7fd7300d1b00 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd72dffb700 1 -- 172.21.15.77:0/289956347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718011d20 con 0x7fd7300d1b00 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 -- 172.21.15.77:0/289956347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300d1b00 msgr2=0x7fd7300d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 --2- 172.21.15.77:0/289956347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300d1b00 0x7fd7300d1f50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd7300cc1f0 tx=0x7fd718009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 -- 172.21.15.77:0/289956347 shutdown_connections 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 --2- 172.21.15.77:0/289956347 >> v2:172.21.15.159:3300/0 conn(0x7fd7300b1cb0 0x7fd7300b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 --2- 172.21.15.77:0/289956347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300d1b00 0x7fd7300d1f50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 -- 172.21.15.77:0/289956347 >> 172.21.15.77:0/289956347 conn(0x7fd7300730a0 msgr2=0x7fd73009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:44.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 -- 172.21.15.77:0/289956347 shutdown_connections 2024-01-15T14:35:44.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.267+0000 7fd735a45700 1 -- 172.21.15.77:0/289956347 wait complete. 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.268+0000 7fd735a45700 1 Processor -- start 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.268+0000 7fd735a45700 1 -- start start 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd735a45700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 0x7fd730156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd735a45700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7300d1b00 0x7fd7300ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd735a45700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd73015e110 0x7fd7300afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd735a45700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c8cf0 con 0x7fd7300b1cb0 2024-01-15T14:35:44.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd735a45700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c8fb0 con 0x7fd73015e110 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd735a45700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c9290 con 0x7fd7300d1b00 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd72effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 0x7fd730156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd72effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 0x7fd730156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40536/0 (socket says 172.21.15.77:40536) 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.269+0000 7fd72effd700 1 -- 172.21.15.77:0/3067781515 learned_addr learned my addr 172.21.15.77:0/3067781515 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72effd700 1 -- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7300d1b00 msgr2=0x7fd7300ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72e7fc700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7300d1b00 0x7fd7300ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72f7fe700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd73015e110 0x7fd7300afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72effd700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7300d1b00 0x7fd7300ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72effd700 1 -- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd73015e110 msgr2=0x7fd7300afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72effd700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd73015e110 0x7fd7300afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72effd700 1 -- 172.21.15.77:0/3067781515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7180094a0 con 0x7fd7300b1cb0 2024-01-15T14:35:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd72effd700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 0x7fd730156a90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c550 tx=0x7fd72800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728004c80 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd728004e00 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728007310 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7300c9570 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.270+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7300c9b30 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.271+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7300c9d50 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.271+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd728007470 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.272+0000 7fd727fff700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd71006b350 0x7fd71006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.272+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd728089870 con 0x7fd7300b1cb0 2024-01-15T14:35:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.272+0000 7fd72e7fc700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd71006b350 0x7fd71006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.272+0000 7fd72e7fc700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd71006b350 0x7fd71006d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd7300b4670 tx=0x7fd718008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:44.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.276+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7280546f0 con 0x7fd7300b1cb0 2024-01-15T14:35:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.482+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd7300751e0 con 0x7fd7300b1cb0 2024-01-15T14:35:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.483+0000 7fd727fff700 1 -- 172.21.15.77:0/3067781515 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd728057d10 con 0x7fd7300b1cb0 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd71006b350 msgr2=0x7fd71006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd71006b350 0x7fd71006d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd7300b4670 tx=0x7fd718008030 comp rx=0 tx=0).stop 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 msgr2=0x7fd730156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 0x7fd730156a90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c550 tx=0x7fd72800c910 comp rx=0 tx=0).stop 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 shutdown_connections 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd71006b350 0x7fd71006d810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd73015e110 0x7fd7300afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7300b1cb0 0x7fd730156a90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 --2- 172.21.15.77:0/3067781515 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd7300d1b00 0x7fd7300ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 >> 172.21.15.77:0/3067781515 conn(0x7fd7300730a0 msgr2=0x7fd7300cfd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 shutdown_connections 2024-01-15T14:35:44.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.485+0000 7fd735a45700 1 -- 172.21.15.77:0/3067781515 wait complete. 2024-01-15T14:35:44.488 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-01-15T14:35:44.498 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-01-15T14:35:44.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:44.499 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-01-15T14:35:44.541 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi159.front.sepia.ceph.com 2024-01-15T14:35:44.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:35:44.541 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T14:35:44.600 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T14:35:44.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:44.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.721+0000 7ff6fe938700 1 Processor -- start 2024-01-15T14:35:44.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.721+0000 7ff6fe938700 1 -- start start 2024-01-15T14:35:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6fe938700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6f815b8d0 0x7ff6f815e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6fe938700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8161b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6fe938700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f80b3520 con 0x7ff6f815bcb0 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6fe938700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80ae230 con 0x7ff6f815b8d0 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6fe938700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80ae490 con 0x7ff6f8071a50 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6f815b8d0 0x7ff6f815e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f77fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff6f815bcb0 0x7ff6f8071340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47314/0 (socket says 172.21.15.77:47314) 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f77fe700 1 -- 172.21.15.77:0/1982000631 learned_addr learned my addr 172.21.15.77:0/1982000631 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6f815b8d0 0x7ff6f815e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40544/0 (socket says 172.21.15.77:40544) 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6fced5700 1 --2- 172.21.15.77:0/1982000631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 -- 172.21.15.77:0/1982000631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 msgr2=0x7ff6f8161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 --2- 172.21.15.77:0/1982000631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8161b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 -- 172.21.15.77:0/1982000631 >> v1:172.21.15.77:6789/0 conn(0x7ff6f815bcb0 legacy=0x7ff6f8071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 -- 172.21.15.77:0/1982000631 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f8162210 con 0x7ff6f815b8d0 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f6ffd700 1 -- 172.21.15.77:0/1982000631 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263037070 0 0) 0x7ff6f80b3520 con 0x7ff6f815bcb0 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f7fff700 1 --2- 172.21.15.77:0/1982000631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815b8d0 0x7ff6f815e520 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009840 tx=0x7ff6e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf9264de8a2d332 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f6ffd700 1 -- 172.21.15.77:0/1982000631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8011830 con 0x7ff6f815b8d0 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f6ffd700 1 -- 172.21.15.77:0/1982000631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8011990 con 0x7ff6f815b8d0 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.722+0000 7ff6f6ffd700 1 -- 172.21.15.77:0/1982000631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8011d20 con 0x7ff6f815b8d0 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 -- 172.21.15.77:0/1982000631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815b8d0 msgr2=0x7ff6f815e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 --2- 172.21.15.77:0/1982000631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815b8d0 0x7ff6f815e520 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009840 tx=0x7ff6e8009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 -- 172.21.15.77:0/1982000631 shutdown_connections 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 --2- 172.21.15.77:0/1982000631 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 --2- 172.21.15.77:0/1982000631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815b8d0 0x7ff6f815e520 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 -- 172.21.15.77:0/1982000631 >> 172.21.15.77:0/1982000631 conn(0x7ff6f80743b0 msgr2=0x7ff6f806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:44.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 -- 172.21.15.77:0/1982000631 shutdown_connections 2024-01-15T14:35:44.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.723+0000 7ff6fe938700 1 -- 172.21.15.77:0/1982000631 wait complete. 2024-01-15T14:35:44.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.724+0000 7ff6fe938700 1 Processor -- start 2024-01-15T14:35:44.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.724+0000 7ff6fe938700 1 -- start start 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fe938700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fe938700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6f815b8d0 0x7ff6f8143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fe938700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 0x7ff6f81585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fe938700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80b2f20 con 0x7ff6f815bcb0 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fe938700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80b2960 con 0x7ff6f815b8d0 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fe938700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80b2c40 con 0x7ff6f8071a50 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 0x7ff6f81585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 0x7ff6f81585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40556/0 (socket says 172.21.15.77:40556) 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 -- 172.21.15.77:0/4007856298 learned_addr learned my addr 172.21.15.77:0/4007856298 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:44.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6f7fff700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6f77fe700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6f815b8d0 0x7ff6f8143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 -- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 msgr2=0x7ff6f8154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 -- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6f815b8d0 msgr2=0x7ff6f8143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6f815b8d0 0x7ff6f8143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.725+0000 7ff6fced5700 1 -- 172.21.15.77:0/4007856298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e80094a0 con 0x7ff6f815bcb0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.726+0000 7ff6fced5700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 0x7ff6f81585d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0007e40 tx=0x7ff6f000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.726+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000ed30 con 0x7ff6f815bcb0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.726+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f0014420 con 0x7ff6f815bcb0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.726+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f8144a60 con 0x7ff6f815bcb0 2024-01-15T14:35:44.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.726+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f00147d0 con 0x7ff6f815bcb0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.726+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f8144490 con 0x7ff6f815bcb0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.727+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f80753f0 con 0x7ff6f815bcb0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.727+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff6f0014930 con 0x7ff6f815bcb0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.727+0000 7ff6f4ff9700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff6e406b350 0x7ff6e406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.728+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6f008a240 con 0x7ff6f815bcb0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.728+0000 7ff6f7fff700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff6e406b350 0x7ff6e406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.728+0000 7ff6f7fff700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff6e406b350 0x7ff6e406d810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80090b0 tx=0x7ff6e8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:44.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.732+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6f00550b0 con 0x7ff6f815bcb0 2024-01-15T14:35:44.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.966+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f8144620 con 0x7ff6f815bcb0 2024-01-15T14:35:44.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.966+0000 7ff6f4ff9700 1 -- 172.21.15.77:0/4007856298 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7ff6f8144620 con 0x7ff6f815bcb0 2024-01-15T14:35:44.967 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:44.968 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:44.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff6e406b350 msgr2=0x7ff6e406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff6e406b350 0x7ff6e406d810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80090b0 tx=0x7ff6e8008030 comp rx=0 tx=0).stop 2024-01-15T14:35:44.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 msgr2=0x7ff6f81585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 0x7ff6f81585d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0007e40 tx=0x7ff6f000e3c0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 shutdown_connections 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff6e406b350 0x7ff6e406d810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff6f8071a50 0x7ff6f8154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6f815bcb0 0x7ff6f81585d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 --2- 172.21.15.77:0/4007856298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff6f815b8d0 0x7ff6f8143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 >> 172.21.15.77:0/4007856298 conn(0x7ff6f80743b0 msgr2=0x7ff6f80aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 shutdown_connections 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:44.968+0000 7ff6fe938700 1 -- 172.21.15.77:0/4007856298 wait complete. 2024-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:44.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:45.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.106+0000 7fd1e47e6700 1 Processor -- start 2024-01-15T14:35:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.106+0000 7fd1e47e6700 1 -- start start 2024-01-15T14:35:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.107+0000 7fd1e47e6700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd1dc147780 0x7fd1dc15e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.107+0000 7fd1e47e6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc15eb00 con 0x7fd1dc06dc20 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.107+0000 7fd1e47e6700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc15ede0 con 0x7fd1dc1473a0 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.107+0000 7fd1e47e6700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1dc15f0c0 con 0x7fd1dc147780 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.107+0000 7fd1e2d83700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd1dc06dc20 0x7fd1dc073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47328/0 (socket says 172.21.15.77:47328) 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.107+0000 7fd1e2d83700 1 -- 172.21.15.77:0/959127640 learned_addr learned my addr 172.21.15.77:0/959127640 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1d81700 1 --2- 172.21.15.77:0/959127640 >> v2:172.21.15.159:3300/0 conn(0x7fd1dc147780 0x7fd1dc15e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574901060 0 0) 0x7fd1dc15eb00 con 0x7fd1dc06dc20 2024-01-15T14:35:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c80035b0 con 0x7fd1dc06dc20 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62808386 0 0) 0x7fd1c80035b0 con 0x7fd1dc06dc20 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 >> v2:172.21.15.159:3300/0 conn(0x7fd1dc147780 msgr2=0x7fd1dc15e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 --2- 172.21.15.77:0/959127640 >> v2:172.21.15.159:3300/0 conn(0x7fd1dc147780 0x7fd1dc15e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 >> v1:172.21.15.159:6789/0 conn(0x7fd1dc1473a0 legacy=0x7fd1dc06c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc15f3a0 con 0x7fd1dc06dc20 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fd1d40019f0 con 0x7fd1dc06dc20 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 >> v1:172.21.15.77:6789/0 conn(0x7fd1dc06dc20 legacy=0x7fd1dc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c8003a00 0x7fd1c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8004850 0x7fd1c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c8005210 0x7fd1c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc15f0c0 con 0x7fd1c8003a00 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c800c0b0 con 0x7fd1c8004850 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c800c270 con 0x7fd1c8005210 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2582700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c8003a00 0x7fd1c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e1d81700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8004850 0x7fd1c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2d83700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c8005210 0x7fd1c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2582700 1 -- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c8005210 msgr2=0x7fd1c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2582700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1c8005210 0x7fd1c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2582700 1 -- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8004850 msgr2=0x7fd1c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2582700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8004850 0x7fd1c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2582700 1 -- 172.21.15.77:0/959127640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c80035b0 con 0x7fd1c8003a00 2024-01-15T14:35:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.108+0000 7fd1e2d83700 1 -- 172.21.15.77:0/959127640 reap_dead start 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e2582700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c8003a00 0x7fd1c8003e20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0017c0 tx=0x7fd1c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babc3454c9316c4e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0052d0 con 0x7fd1c8003a00 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc005450 con 0x7fd1c8003a00 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e1580700 1 -- 172.21.15.77:0/959127640 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0038c0 con 0x7fd1c8003a00 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 -- 172.21.15.77:0/959127640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c8003a00 msgr2=0x7fd1c8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c8003a00 0x7fd1c8003e20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0017c0 tx=0x7fd1c8005880 comp rx=0 tx=0).stop 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 -- 172.21.15.77:0/959127640 shutdown_connections 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/959127640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c8003a00 0x7fd1c8003e20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 -- 172.21.15.77:0/959127640 >> 172.21.15.77:0/959127640 conn(0x7fd1dc0b3130 msgr2=0x7fd1dc0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 -- 172.21.15.77:0/959127640 shutdown_connections 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.109+0000 7fd1e47e6700 1 -- 172.21.15.77:0/959127640 wait complete. 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 Processor -- start 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 -- start start 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8003a00 0x7fd1dc0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc1f6fa0 0x7fd1dc1f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 0x7fd1dc1344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1f7960 con 0x7fd1dc134000 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0ad870 con 0x7fd1c8003a00 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e47e6700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0adb50 con 0x7fd1dc1f6fa0 2024-01-15T14:35:45.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e2d83700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 0x7fd1dc1344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e2d83700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 0x7fd1dc1344b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40566/0 (socket says 172.21.15.77:40566) 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.111+0000 7fd1e2d83700 1 -- 172.21.15.77:0/4015552463 learned_addr learned my addr 172.21.15.77:0/4015552463 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e1d81700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc1f6fa0 0x7fd1dc1f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2582700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8003a00 0x7fd1dc0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2d83700 1 -- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc1f6fa0 msgr2=0x7fd1dc1f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2d83700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc1f6fa0 0x7fd1dc1f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2d83700 1 -- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8003a00 msgr2=0x7fd1dc0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2d83700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8003a00 0x7fd1dc0aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2d83700 1 -- 172.21.15.77:0/4015552463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc0ade30 con 0x7fd1dc134000 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e1d81700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc1f6fa0 0x7fd1dc1f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e2d83700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 0x7fd1dc1344b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0b46f0 tx=0x7fd1d4007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d400ebe0 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d400ed40 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc0ae110 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4013660 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.112+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc0ae6d0 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.113+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1dc069000 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.113+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd1d40137c0 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.114+0000 7fd1d2ffd700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd1c006b350 0x7fd1c006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.114+0000 7fd1e2582700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd1c006b350 0x7fd1c006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.114+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1d408a100 con 0x7fd1dc134000 2024-01-15T14:35:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.114+0000 7fd1e2582700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd1c006b350 0x7fd1c006d810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc015040 tx=0x7fd1cc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.119+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1d4054f70 con 0x7fd1dc134000 2024-01-15T14:35:45.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.321+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1dc067f60 con 0x7fd1dc134000 2024-01-15T14:35:45.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.322+0000 7fd1d2ffd700 1 -- 172.21.15.77:0/4015552463 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16753 (secure 0 0 0) 0x7fd1d4058590 con 0x7fd1dc134000 2024-01-15T14:35:45.323 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:45.323 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":22,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:45.229066+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd1c006b350 msgr2=0x7fd1c006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd1c006b350 0x7fd1c006d810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc015040 tx=0x7fd1cc009030 comp rx=0 tx=0).stop 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 msgr2=0x7fd1dc1344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 0x7fd1dc1344b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0b46f0 tx=0x7fd1d4007c90 comp rx=0 tx=0).stop 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 shutdown_connections 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd1c006b350 0x7fd1c006d810 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1dc134000 0x7fd1dc1344b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd1c8003a00 0x7fd1dc0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 --2- 172.21.15.77:0/4015552463 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd1dc1f6fa0 0x7fd1dc1f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 >> 172.21.15.77:0/4015552463 conn(0x7fd1dc0b3130 msgr2=0x7fd1dc06b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:45.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 shutdown_connections 2024-01-15T14:35:45.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.324+0000 7fd1e47e6700 1 -- 172.21.15.77:0/4015552463 wait complete. 2024-01-15T14:35:45.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T14:35:02.432547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T14:35:03.553935+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:45.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:45.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.462+0000 7f385dd06700 1 Processor -- start 2024-01-15T14:35:45.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.462+0000 7f385dd06700 1 -- start start 2024-01-15T14:35:45.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f385dd06700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f385815e110 0x7f385806f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f385dd06700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f385806e420 0x7f385806e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f385dd06700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385806d980 con 0x7f38580afb30 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f385dd06700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38580affb0 con 0x7f385806e420 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f385dd06700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38580b0260 con 0x7f385815e110 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3856ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f38580afb30 0x7f385806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47336/0 (socket says 172.21.15.77:47336) 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3857fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f385806e420 0x7f385806e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3856ffd700 1 -- 172.21.15.77:0/587381096 learned_addr learned my addr 172.21.15.77:0/587381096 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f38577fe700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f385815e110 0x7f385806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3857fff700 1 -- 172.21.15.77:0/587381096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f385815e110 msgr2=0x7f385806f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3857fff700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f385815e110 0x7f385806f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3857fff700 1 -- 172.21.15.77:0/587381096 >> v1:172.21.15.77:6789/0 conn(0x7f38580afb30 legacy=0x7f385806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3857fff700 1 -- 172.21.15.77:0/587381096 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38580b0540 con 0x7f385806e420 2024-01-15T14:35:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f38577fe700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f385815e110 0x7f385806f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f3857fff700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385806e420 0x7f385806e8b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3850009d90 tx=0x7f385000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c2cc7b50f2a212 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f38567fc700 1 -- 172.21.15.77:0/587381096 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385000e070 con 0x7f385806e420 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f38567fc700 1 -- 172.21.15.77:0/587381096 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385000bbd0 con 0x7f385806e420 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.463+0000 7f38567fc700 1 -- 172.21.15.77:0/587381096 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850012430 con 0x7f385806e420 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 -- 172.21.15.77:0/587381096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385806e420 msgr2=0x7f385806e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385806e420 0x7f385806e8b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3850009d90 tx=0x7f385000b330 comp rx=0 tx=0).stop 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 -- 172.21.15.77:0/587381096 shutdown_connections 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385806e420 0x7f385806e8b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 --2- 172.21.15.77:0/587381096 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f385815e110 0x7f385806f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 -- 172.21.15.77:0/587381096 >> 172.21.15.77:0/587381096 conn(0x7f38580b3250 msgr2=0x7f38580d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 -- 172.21.15.77:0/587381096 shutdown_connections 2024-01-15T14:35:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.464+0000 7f385dd06700 1 -- 172.21.15.77:0/587381096 wait complete. 2024-01-15T14:35:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.465+0000 7f385dd06700 1 Processor -- start 2024-01-15T14:35:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.465+0000 7f385dd06700 1 -- start start 2024-01-15T14:35:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f385dd06700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f385806e420 0x7f385814afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f385dd06700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38580afb30 0x7f38580ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f385dd06700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 0x7f38580ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f385dd06700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385814b520 con 0x7f385815e110 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f385dd06700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385814b7c0 con 0x7f385806e420 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f385dd06700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385814c340 con 0x7f38580afb30 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 0x7f38580ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 0x7f38580ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40580/0 (socket says 172.21.15.77:40580) 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 -- 172.21.15.77:0/784499049 learned_addr learned my addr 172.21.15.77:0/784499049 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f38577fe700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f385806e420 0x7f385814afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3856ffd700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38580afb30 0x7f38580ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 -- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38580afb30 msgr2=0x7f38580ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38580afb30 0x7f38580ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 -- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f385806e420 msgr2=0x7f385814afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.466+0000 7f3857fff700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f385806e420 0x7f385814afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f3857fff700 1 -- 172.21.15.77:0/784499049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850009a70 con 0x7f385815e110 2024-01-15T14:35:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f3856ffd700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38580afb30 0x7f38580ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f38577fe700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f385806e420 0x7f385814afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f3857fff700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 0x7f38580ca810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3850014040 tx=0x7f3850004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38500098f0 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3850020070 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385806b880 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850004280 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.467+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385814bcf0 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.468+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3858069020 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.468+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f38500043e0 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.468+0000 7f383ffff700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f384406b380 0x7f384406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.469+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3850022070 con 0x7f385815e110 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.469+0000 7f38577fe700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f384406b380 0x7f384406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.469+0000 7f38577fe700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f384406b380 0x7f384406d840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f384800aa20 tx=0x7f3848008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.473+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3850056920 con 0x7f385815e110 2024-01-15T14:35:45.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.660+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f385814c480 con 0x7f385815e110 2024-01-15T14:35:45.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.660+0000 7f383ffff700 1 -- 172.21.15.77:0/784499049 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f385002c020 con 0x7f385815e110 2024-01-15T14:35:45.662 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:45.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f384406b380 msgr2=0x7f384406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f384406b380 0x7f384406d840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f384800aa20 tx=0x7f3848008040 comp rx=0 tx=0).stop 2024-01-15T14:35:45.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 msgr2=0x7f38580ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 0x7f38580ca810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3850014040 tx=0x7f3850004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 shutdown_connections 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f384406b380 0x7f384406d840 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f385815e110 0x7f38580ca810 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f385806e420 0x7f385814afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 --2- 172.21.15.77:0/784499049 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f38580afb30 0x7f38580ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 >> 172.21.15.77:0/784499049 conn(0x7f38580b3250 msgr2=0x7f38580d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.662+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 shutdown_connections 2024-01-15T14:35:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.663+0000 7f385dd06700 1 -- 172.21.15.77:0/784499049 wait complete. 2024-01-15T14:35:45.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T14:35:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.799+0000 7efef1e0b700 1 Processor -- start 2024-01-15T14:35:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.799+0000 7efef1e0b700 1 -- start start 2024-01-15T14:35:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efef1e0b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efeec161690 0x7efeec0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efef1e0b700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efeec09b110 0x7efeec0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efef1e0b700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0d0a70 con 0x7efeec161a70 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efef1e0b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0d1c10 con 0x7efeec161690 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efef1e0b700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0b2ab0 con 0x7efeec09b110 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efeec161690 0x7efeec0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efeec161690 0x7efeec0d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40588/0 (socket says 172.21.15.77:40588) 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 -- 172.21.15.77:0/2324648669 learned_addr learned my addr 172.21.15.77:0/2324648669 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeebfff700 1 --2- 172.21.15.77:0/2324648669 >> v2:172.21.15.159:3300/0 conn(0x7efeec09b110 0x7efeec0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 -- 172.21.15.77:0/2324648669 >> v2:172.21.15.159:3300/0 conn(0x7efeec09b110 msgr2=0x7efeec0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 --2- 172.21.15.77:0/2324648669 >> v2:172.21.15.159:3300/0 conn(0x7efeec09b110 0x7efeec0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 -- 172.21.15.77:0/2324648669 >> v1:172.21.15.159:6789/0 conn(0x7efeec161a70 legacy=0x7efeec09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 -- 172.21.15.77:0/2324648669 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0b2d90 con 0x7efeec161690 2024-01-15T14:35:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.800+0000 7efeeb7fe700 1 --2- 172.21.15.77:0/2324648669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec161690 0x7efeec0d0f20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efeec0ce460 tx=0x7efed4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4bdfceeb765099 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efeea7fc700 1 -- 172.21.15.77:0/2324648669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4004230 con 0x7efeec161690 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efeea7fc700 1 -- 172.21.15.77:0/2324648669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed4004390 con 0x7efeec161690 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efeea7fc700 1 -- 172.21.15.77:0/2324648669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed40036a0 con 0x7efeec161690 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 -- 172.21.15.77:0/2324648669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec161690 msgr2=0x7efeec0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 --2- 172.21.15.77:0/2324648669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec161690 0x7efeec0d0f20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efeec0ce460 tx=0x7efed4009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 -- 172.21.15.77:0/2324648669 shutdown_connections 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 --2- 172.21.15.77:0/2324648669 >> v2:172.21.15.159:3300/0 conn(0x7efeec09b110 0x7efeec0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 --2- 172.21.15.77:0/2324648669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec161690 0x7efeec0d0f20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 -- 172.21.15.77:0/2324648669 >> 172.21.15.77:0/2324648669 conn(0x7efeec0d1e50 msgr2=0x7efeec0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 -- 172.21.15.77:0/2324648669 shutdown_connections 2024-01-15T14:35:45.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.801+0000 7efef1e0b700 1 -- 172.21.15.77:0/2324648669 wait complete. 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.802+0000 7efef1e0b700 1 Processor -- start 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 -- start start 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 0x7efeec15c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeec161690 0x7efeec1f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeec161a70 0x7efeec1f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec1f7420 con 0x7efeec09b110 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec1f7680 con 0x7efeec161a70 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efef1e0b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec1f7960 con 0x7efeec161690 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efeeb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 0x7efeec15c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efeeb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 0x7efeec15c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40600/0 (socket says 172.21.15.77:40600) 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.803+0000 7efeeb7fe700 1 -- 172.21.15.77:0/582981067 learned_addr learned my addr 172.21.15.77:0/582981067 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeebfff700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeec161a70 0x7efeec1f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeaffd700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeec161690 0x7efeec1f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeb7fe700 1 -- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeec161690 msgr2=0x7efeec1f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeb7fe700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeec161690 0x7efeec1f6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeb7fe700 1 -- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeec161a70 msgr2=0x7efeec1f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeb7fe700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeec161a70 0x7efeec1f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeb7fe700 1 -- 172.21.15.77:0/582981067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed4009550 con 0x7efeec09b110 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeaffd700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeec161690 0x7efeec1f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efeeb7fe700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 0x7efeec15c850 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efeec072870 tx=0x7efed4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4003de0 con 0x7efeec09b110 2024-01-15T14:35:45.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed4011630 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec1f7c40 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed40119c0 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.804+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec1f3bf0 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.805+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec062220 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.805+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efed4011b20 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.806+0000 7efee3fff700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efecc06b350 0x7efecc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.806+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efed408b1e0 con 0x7efeec09b110 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.806+0000 7efeeaffd700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efecc06b350 0x7efecc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:45.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.806+0000 7efeeaffd700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efecc06b350 0x7efecc06d810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efeec1f6e40 tx=0x7efedc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:45.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:45.810+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efed4055fe0 con 0x7efeec09b110 2024-01-15T14:35:46.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.001+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efeec069020 con 0x7efeec09b110 2024-01-15T14:35:46.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.002+0000 7efee3fff700 1 -- 172.21.15.77:0/582981067 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7efed4059600 con 0x7efeec09b110 2024-01-15T14:35:46.003 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efecc06b350 msgr2=0x7efecc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efecc06b350 0x7efecc06d810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efeec1f6e40 tx=0x7efedc009040 comp rx=0 tx=0).stop 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 msgr2=0x7efeec15c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 0x7efeec15c850 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efeec072870 tx=0x7efed4004750 comp rx=0 tx=0).stop 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 shutdown_connections 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efecc06b350 0x7efecc06d810 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efeec161a70 0x7efeec1f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeec09b110 0x7efeec15c850 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 --2- 172.21.15.77:0/582981067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efeec161690 0x7efeec1f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 >> 172.21.15.77:0/582981067 conn(0x7efeec0d1e50 msgr2=0x7efeec0d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 shutdown_connections 2024-01-15T14:35:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.004+0000 7efef1e0b700 1 -- 172.21.15.77:0/582981067 wait complete. 2024-01-15T14:35:46.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T14:35:46.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.141+0000 7f5f21758700 1 Processor -- start 2024-01-15T14:35:46.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.141+0000 7f5f21758700 1 -- start start 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f21758700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f1c15ad60 0x7f5f1c0af540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f21758700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f21758700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0b4a10 con 0x7f5f1c06c000 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f21758700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1c0b3e00 con 0x7f5f1c15ad60 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f21758700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c0b40e0 con 0x7f5f1c0ae730 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f1c15ad60 0x7f5f1c0af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f1c15ad60 0x7f5f1c0af540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40608/0 (socket says 172.21.15.77:40608) 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1b7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5f1c06c000 0x7f5f1c06dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47350/0 (socket says 172.21.15.77:47350) 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1affd700 1 -- 172.21.15.77:0/2473619135 learned_addr learned my addr 172.21.15.77:0/2473619135 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1a7fc700 1 --2- 172.21.15.77:0/2473619135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1affd700 1 -- 172.21.15.77:0/2473619135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 msgr2=0x7f5f1c0b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1affd700 1 --2- 172.21.15.77:0/2473619135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c0b4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.142+0000 7f5f1affd700 1 -- 172.21.15.77:0/2473619135 >> v1:172.21.15.77:6789/0 conn(0x7f5f1c06c000 legacy=0x7f5f1c06dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f1affd700 1 -- 172.21.15.77:0/2473619135 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c0b43c0 con 0x7f5f1c15ad60 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f19ffb700 1 -- 172.21.15.77:0/2473619135 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899016689 0 0) 0x7f5f1c0b4a10 con 0x7f5f1c06c000 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f1affd700 1 --2- 172.21.15.77:0/2473619135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c0af540 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5f100098f0 tx=0x7f5f10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9d1b487e63d3c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f19ffb700 1 -- 172.21.15.77:0/2473619135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1000b8a0 con 0x7f5f1c15ad60 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f19ffb700 1 -- 172.21.15.77:0/2473619135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1000ba00 con 0x7f5f1c15ad60 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f19ffb700 1 -- 172.21.15.77:0/2473619135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1000bd90 con 0x7f5f1c15ad60 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 -- 172.21.15.77:0/2473619135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c15ad60 msgr2=0x7f5f1c0af540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 --2- 172.21.15.77:0/2473619135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c0af540 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5f100098f0 tx=0x7f5f10009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 -- 172.21.15.77:0/2473619135 shutdown_connections 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 --2- 172.21.15.77:0/2473619135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c0af540 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 --2- 172.21.15.77:0/2473619135 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c0b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 -- 172.21.15.77:0/2473619135 >> 172.21.15.77:0/2473619135 conn(0x7f5f1c071bb0 msgr2=0x7f5f1c071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 -- 172.21.15.77:0/2473619135 shutdown_connections 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.143+0000 7f5f21758700 1 -- 172.21.15.77:0/2473619135 wait complete. 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 Processor -- start 2024-01-15T14:35:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 -- start start 2024-01-15T14:35:46.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 0x7f5f1c09dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c1425c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c142b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c143040 con 0x7f5f1c06c000 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c09cb70 con 0x7f5f1c15ad60 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.145+0000 7f5f21758700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c09ce50 con 0x7f5f1c0ae730 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 0x7f5f1c09dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 0x7f5f1c09dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40616/0 (socket says 172.21.15.77:40616) 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 -- 172.21.15.77:0/2625638979 learned_addr learned my addr 172.21.15.77:0/2625638979 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1a7fc700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c1425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1b7fe700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c142b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 -- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 msgr2=0x7f5f1c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c1425c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 -- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f1c15ad60 msgr2=0x7f5f1c142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c142b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 -- 172.21.15.77:0/2625638979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10009550 con 0x7f5f1c06c000 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f1affd700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 0x7f5f1c09dc10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009c40 tx=0x7f5f10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f100045d0 con 0x7f5f1c06c000 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1001e070 con 0x7f5f1c06c000 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c0b5560 con 0x7f5f1c06c000 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f100038e0 con 0x7f5f1c06c000 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.146+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c09d410 con 0x7f5f1c06c000 2024-01-15T14:35:46.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.147+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c069020 con 0x7f5f1c06c000 2024-01-15T14:35:46.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.148+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5f10003a40 con 0x7f5f1c06c000 2024-01-15T14:35:46.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.148+0000 7f5f037fe700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f0806b350 0x7f5f0806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.148+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5f10020070 con 0x7f5f1c06c000 2024-01-15T14:35:46.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.148+0000 7f5f1a7fc700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f0806b350 0x7f5f0806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.148+0000 7f5f1a7fc700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f0806b350 0x7f5f0806d810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c146dc0 tx=0x7f5f0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.152+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f10055f40 con 0x7f5f1c06c000 2024-01-15T14:35:46.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.350+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f1c06fa50 con 0x7f5f1c06c000 2024-01-15T14:35:46.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.351+0000 7f5f037fe700 1 -- 172.21.15.77:0/2625638979 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f1002ee80 con 0x7f5f1c06c000 2024-01-15T14:35:46.352 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f0806b350 msgr2=0x7f5f0806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f0806b350 0x7f5f0806d810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c146dc0 tx=0x7f5f0c008040 comp rx=0 tx=0).stop 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 msgr2=0x7f5f1c09dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 0x7f5f1c09dc10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009c40 tx=0x7f5f10004750 comp rx=0 tx=0).stop 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 shutdown_connections 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f0806b350 0x7f5f0806d810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f1c15ad60 0x7f5f1c142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f1c06c000 0x7f5f1c09dc10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 --2- 172.21.15.77:0/2625638979 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f1c0ae730 0x7f5f1c1425c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 >> 172.21.15.77:0/2625638979 conn(0x7f5f1c071bb0 msgr2=0x7f5f1c06e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:46.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 shutdown_connections 2024-01-15T14:35:46.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.354+0000 7f5f21758700 1 -- 172.21.15.77:0/2625638979 wait complete. 2024-01-15T14:35:46.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:46.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 Processor -- start 2024-01-15T14:35:46.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 -- start start 2024-01-15T14:35:46.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c147780 0x7ff49c15e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff49c06dc20 0x7ff49c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c15eb00 con 0x7ff49c1473a0 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c15ede0 con 0x7ff49c06dc20 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff4a1b01700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c15f0c0 con 0x7ff49c147780 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff49c06dc20 0x7ff49c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff49c06dc20 0x7ff49c073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40630/0 (socket says 172.21.15.77:40630) 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 -- 172.21.15.77:0/600991494 learned_addr learned my addr 172.21.15.77:0/600991494 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 -- 172.21.15.77:0/600991494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c147780 msgr2=0x7ff49c15e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49affd700 1 --2- 172.21.15.77:0/600991494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c147780 0x7ff49c15e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 --2- 172.21.15.77:0/600991494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c147780 0x7ff49c15e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 -- 172.21.15.77:0/600991494 >> v1:172.21.15.159:6789/0 conn(0x7ff49c1473a0 legacy=0x7ff49c06c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 -- 172.21.15.77:0/600991494 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c15f3a0 con 0x7ff49c06dc20 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49bfff700 1 --2- 172.21.15.77:0/600991494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c06dc20 0x7ff49c073f90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff494009e50 tx=0x7ff49400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f223802c2a1fb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49a7fc700 1 -- 172.21.15.77:0/600991494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49400e070 con 0x7ff49c06dc20 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49a7fc700 1 -- 172.21.15.77:0/600991494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff494003a20 con 0x7ff49c06dc20 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.487+0000 7ff49a7fc700 1 -- 172.21.15.77:0/600991494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494003dd0 con 0x7ff49c06dc20 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 -- 172.21.15.77:0/600991494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c06dc20 msgr2=0x7ff49c073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/600991494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c06dc20 0x7ff49c073f90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff494009e50 tx=0x7ff49400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 -- 172.21.15.77:0/600991494 shutdown_connections 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/600991494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c06dc20 0x7ff49c073f90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/600991494 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c147780 0x7ff49c15e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 -- 172.21.15.77:0/600991494 >> 172.21.15.77:0/600991494 conn(0x7ff49c0b3130 msgr2=0x7ff49c0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:46.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 -- 172.21.15.77:0/600991494 shutdown_connections 2024-01-15T14:35:46.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.488+0000 7ff4a1b01700 1 -- 172.21.15.77:0/600991494 wait complete. 2024-01-15T14:35:46.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.489+0000 7ff4a1b01700 1 Processor -- start 2024-01-15T14:35:46.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.489+0000 7ff4a1b01700 1 -- start start 2024-01-15T14:35:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff4a1b01700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff49c06dc20 0x7ff49c15ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff4a1b01700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c1473a0 0x7ff49c144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff4a1b01700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 0x7ff49c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff4a1b01700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c143450 con 0x7ff49c147780 2024-01-15T14:35:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff4a1b01700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c15d3b0 con 0x7ff49c06dc20 2024-01-15T14:35:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff4a1b01700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c143040 con 0x7ff49c1473a0 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff49bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 0x7ff49c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff49bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 0x7ff49c06ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40632/0 (socket says 172.21.15.77:40632) 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff49bfff700 1 -- 172.21.15.77:0/1513415613 learned_addr learned my addr 172.21.15.77:0/1513415613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff49b7fe700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff49c06dc20 0x7ff49c15ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.490+0000 7ff49affd700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c1473a0 0x7ff49c144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49bfff700 1 -- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c1473a0 msgr2=0x7ff49c144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49bfff700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c1473a0 0x7ff49c144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49bfff700 1 -- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff49c06dc20 msgr2=0x7ff49c15ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49bfff700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff49c06dc20 0x7ff49c15ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49bfff700 1 -- 172.21.15.77:0/1513415613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494009b30 con 0x7ff49c147780 2024-01-15T14:35:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49b7fe700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff49c06dc20 0x7ff49c15ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff49bfff700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 0x7ff49c06ec10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff49c074d90 tx=0x7ff494004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494004440 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4940045a0 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c1436a0 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494004950 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.491+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c143c60 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.492+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c067f60 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.492+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff494025450 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.493+0000 7ff483fff700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff48806b350 0x7ff48806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.493+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff49408b500 con 0x7ff49c147780 2024-01-15T14:35:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.493+0000 7ff49b7fe700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff48806b350 0x7ff48806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.493+0000 7ff49b7fe700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff48806b350 0x7ff48806d810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff48c003fc0 tx=0x7ff48c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.497+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff494056300 con 0x7ff49c147780 2024-01-15T14:35:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.728+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49c06c240 con 0x7ff49c147780 2024-01-15T14:35:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.729+0000 7ff483fff700 1 -- 172.21.15.77:0/1513415613 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7ff494059920 con 0x7ff49c147780 2024-01-15T14:35:46.730 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff48806b350 msgr2=0x7ff48806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff48806b350 0x7ff48806d810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff48c003fc0 tx=0x7ff48c009040 comp rx=0 tx=0).stop 2024-01-15T14:35:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 msgr2=0x7ff49c06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 0x7ff49c06ec10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff49c074d90 tx=0x7ff494004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 shutdown_connections 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff48806b350 0x7ff48806d810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff49c147780 0x7ff49c06ec10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff49c06dc20 0x7ff49c15ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 --2- 172.21.15.77:0/1513415613 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff49c1473a0 0x7ff49c144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 >> 172.21.15.77:0/1513415613 conn(0x7ff49c0b3130 msgr2=0x7ff49c06e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 shutdown_connections 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.731+0000 7ff4a1b01700 1 -- 172.21.15.77:0/1513415613 wait complete. 2024-01-15T14:35:46.734 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:46.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:46.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 Processor -- start 2024-01-15T14:35:46.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 -- start start 2024-01-15T14:35:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d640d45d0 0x7f8d640b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d640d39d0 0x7f8d640b7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640d2930 con 0x7f8d6415e140 2024-01-15T14:35:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6415e520 con 0x7f8d640d45d0 2024-01-15T14:35:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.866+0000 7f8d69722700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d2c10 con 0x7f8d640d39d0 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d627fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d640d45d0 0x7f8d640b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d62ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8d6415e140 0x7f8d640d3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47352/0 (socket says 172.21.15.77:47352) 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d62ffd700 1 -- 172.21.15.77:0/2777350028 learned_addr learned my addr 172.21.15.77:0/2777350028 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d637fe700 1 --2- 172.21.15.77:0/2777350028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d640d39d0 0x7f8d640b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d627fc700 1 -- 172.21.15.77:0/2777350028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d640d39d0 msgr2=0x7f8d640b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d627fc700 1 --2- 172.21.15.77:0/2777350028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d640d39d0 0x7f8d640b7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d627fc700 1 -- 172.21.15.77:0/2777350028 >> v1:172.21.15.77:6789/0 conn(0x7f8d6415e140 legacy=0x7f8d640d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d627fc700 1 -- 172.21.15.77:0/2777350028 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640d2ef0 con 0x7f8d640d45d0 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d61ffb700 1 -- 172.21.15.77:0/2777350028 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727478809 0 0) 0x7f8d640d2930 con 0x7f8d6415e140 2024-01-15T14:35:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d627fc700 1 --2- 172.21.15.77:0/2777350028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d45d0 0x7f8d640b70d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8d6415e660 tx=0x7f8d54009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8877c3f2bbe6131f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d61ffb700 1 -- 172.21.15.77:0/2777350028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54004020 con 0x7f8d640d45d0 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d61ffb700 1 -- 172.21.15.77:0/2777350028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d54004180 con 0x7f8d640d45d0 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.867+0000 7f8d61ffb700 1 -- 172.21.15.77:0/2777350028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54004530 con 0x7f8d640d45d0 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 -- 172.21.15.77:0/2777350028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d45d0 msgr2=0x7f8d640b70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 --2- 172.21.15.77:0/2777350028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d45d0 0x7f8d640b70d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8d6415e660 tx=0x7f8d54009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 -- 172.21.15.77:0/2777350028 shutdown_connections 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 --2- 172.21.15.77:0/2777350028 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d640d39d0 0x7f8d640b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 --2- 172.21.15.77:0/2777350028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d45d0 0x7f8d640b70d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 -- 172.21.15.77:0/2777350028 >> 172.21.15.77:0/2777350028 conn(0x7f8d6406b410 msgr2=0x7f8d6406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 -- 172.21.15.77:0/2777350028 shutdown_connections 2024-01-15T14:35:46.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.868+0000 7f8d69722700 1 -- 172.21.15.77:0/2777350028 wait complete. 2024-01-15T14:35:46.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.869+0000 7f8d69722700 1 Processor -- start 2024-01-15T14:35:46.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.869+0000 7f8d69722700 1 -- start start 2024-01-15T14:35:46.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d69722700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 0x7f8d640cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d69722700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d640d45d0 0x7f8d640b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d69722700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d6415e140 0x7f8d640b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d69722700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d641f4730 con 0x7f8d640d39d0 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d69722700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d641f4450 con 0x7f8d640d45d0 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d69722700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640b3b10 con 0x7f8d6415e140 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 0x7f8d640cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 0x7f8d640cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40652/0 (socket says 172.21.15.77:40652) 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 -- 172.21.15.77:0/3041907287 learned_addr learned my addr 172.21.15.77:0/3041907287 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d627fc700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d640d45d0 0x7f8d640b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d637fe700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d6415e140 0x7f8d640b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 -- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d6415e140 msgr2=0x7f8d640b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d6415e140 0x7f8d640b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 -- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d640d45d0 msgr2=0x7f8d640b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:46.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d640d45d0 0x7f8d640b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.870+0000 7f8d62ffd700 1 -- 172.21.15.77:0/3041907287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d540094a0 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.871+0000 7f8d62ffd700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 0x7f8d640cb000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00c550 tx=0x7f8d4c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.871+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c004c80 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.871+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c004e00 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.871+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d640b3df0 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.871+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c007310 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.871+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6415c120 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.872+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d640b3970 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.872+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8d4c007470 con 0x7f8d640d39d0 2024-01-15T14:35:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.872+0000 7f8d5b7fe700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d4406b350 0x7f8d4406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:46.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.873+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8d4c089850 con 0x7f8d640d39d0 2024-01-15T14:35:46.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.873+0000 7f8d627fc700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d4406b350 0x7f8d4406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:46.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.873+0000 7f8d627fc700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d4406b350 0x7f8d4406d810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8d6415e660 tx=0x7f8d54008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:46.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:46.878+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d640b3970 con 0x7f8d640d39d0 2024-01-15T14:35:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.073+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d640b3970 con 0x7f8d640d39d0 2024-01-15T14:35:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.074+0000 7f8d5b7fe700 1 -- 172.21.15.77:0/3041907287 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16753 (secure 0 0 0) 0x7f8d640b3970 con 0x7f8d640d39d0 2024-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:46.233118+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d4406b350 msgr2=0x7f8d4406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d4406b350 0x7f8d4406d810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8d6415e660 tx=0x7f8d54008030 comp rx=0 tx=0).stop 2024-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 msgr2=0x7f8d640cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 0x7f8d640cb000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00c550 tx=0x7f8d4c00c860 comp rx=0 tx=0).stop 2024-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 shutdown_connections 2024-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8d4406b350 0x7f8d4406d810 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8d6415e140 0x7f8d640b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d640d39d0 0x7f8d640cb000 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 --2- 172.21.15.77:0/3041907287 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8d640d45d0 0x7f8d640b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 >> 172.21.15.77:0/3041907287 conn(0x7f8d6406b410 msgr2=0x7f8d640d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 shutdown_connections 2024-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.076+0000 7f8d69722700 1 -- 172.21.15.77:0/3041907287 wait complete. 2024-01-15T14:35:47.089 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-15T14:35:47.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:47.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.210+0000 7fc2e8f44700 1 Processor -- start 2024-01-15T14:35:47.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.210+0000 7fc2e8f44700 1 -- start start 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e8f44700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc2e415bdd0 0x7fc2e41604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e8f44700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e8f44700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40b2030 con 0x7fc2e40d1c70 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e8f44700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e40b2f50 con 0x7fc2e415bdd0 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e8f44700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40b22d0 con 0x7fc2e40b3370 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc2e415bdd0 0x7fc2e41604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e2d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc2e40d1c70 0x7fc2e40ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47364/0 (socket says 172.21.15.77:47364) 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e2d9d700 1 -- 172.21.15.77:0/708367502 learned_addr learned my addr 172.21.15.77:0/708367502 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:47.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e1d9b700 1 --2- 172.21.15.77:0/708367502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e259c700 1 -- 172.21.15.77:0/708367502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 msgr2=0x7fc2e40b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e259c700 1 --2- 172.21.15.77:0/708367502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e159a700 1 -- 172.21.15.77:0/708367502 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601381409 0 0) 0x7fc2e40b2030 con 0x7fc2e40d1c70 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e259c700 1 -- 172.21.15.77:0/708367502 >> v1:172.21.15.77:6789/0 conn(0x7fc2e40d1c70 legacy=0x7fc2e40ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e259c700 1 -- 172.21.15.77:0/708367502 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40b25b0 con 0x7fc2e415bdd0 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.211+0000 7fc2e259c700 1 --2- 172.21.15.77:0/708367502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e415bdd0 0x7fc2e41604e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009840 tx=0x7fc2d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480a39f40e9ded06 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e159a700 1 -- 172.21.15.77:0/708367502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011830 con 0x7fc2e415bdd0 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e159a700 1 -- 172.21.15.77:0/708367502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4011990 con 0x7fc2e415bdd0 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e159a700 1 -- 172.21.15.77:0/708367502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011d20 con 0x7fc2e415bdd0 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 -- 172.21.15.77:0/708367502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e415bdd0 msgr2=0x7fc2e41604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/708367502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e415bdd0 0x7fc2e41604e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009840 tx=0x7fc2d4009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 -- 172.21.15.77:0/708367502 shutdown_connections 2024-01-15T14:35:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/708367502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e415bdd0 0x7fc2e41604e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/708367502 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 -- 172.21.15.77:0/708367502 >> 172.21.15.77:0/708367502 conn(0x7fc2e40731a0 msgr2=0x7fc2e409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:47.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 -- 172.21.15.77:0/708367502 shutdown_connections 2024-01-15T14:35:47.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.212+0000 7fc2e8f44700 1 -- 172.21.15.77:0/708367502 wait complete. 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.213+0000 7fc2e8f44700 1 Processor -- start 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 -- start start 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 0x7fc2e4151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2e415bdd0 0x7fc2e40af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40ca2d0 con 0x7fc2e40d1c70 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40c5d30 con 0x7fc2e415bdd0 2024-01-15T14:35:47.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e8f44700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41f2880 con 0x7fc2e40b3370 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 0x7fc2e4151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 0x7fc2e4151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40660/0 (socket says 172.21.15.77:40660) 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.214+0000 7fc2e1d9b700 1 -- 172.21.15.77:0/3775988492 learned_addr learned my addr 172.21.15.77:0/3775988492 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e2d9d700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2e415bdd0 0x7fc2e40af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e259c700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e1d9b700 1 -- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 msgr2=0x7fc2e40c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e1d9b700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40c57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e1d9b700 1 -- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2e415bdd0 msgr2=0x7fc2e40af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e1d9b700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2e415bdd0 0x7fc2e40af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e1d9b700 1 -- 172.21.15.77:0/3775988492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40094a0 con 0x7fc2e40d1c70 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e2d9d700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2e415bdd0 0x7fc2e40af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e1d9b700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 0x7fc2e4151200 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800c7f0 tx=0x7fc2d800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8011800 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8011960 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e40ca590 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8011cf0 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.215+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e40cab50 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.216+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4067f60 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.216+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc2d8018360 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.217+0000 7fc2d2ffd700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2c406b350 0x7fc2c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.217+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc2d8089be0 con 0x7fc2e40d1c70 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.217+0000 7fc2e259c700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2c406b350 0x7fc2c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.217+0000 7fc2e259c700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2c406b350 0x7fc2c406d810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40c5170 tx=0x7fc2d4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.221+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2d80549e0 con 0x7fc2e40d1c70 2024-01-15T14:35:47.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.454+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2e40cfd70 con 0x7fc2e40d1c70 2024-01-15T14:35:47.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.454+0000 7fc2d2ffd700 1 -- 172.21.15.77:0/3775988492 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fc2d8058000 con 0x7fc2e40d1c70 2024-01-15T14:35:47.454 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:47.454 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2c406b350 msgr2=0x7fc2c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2c406b350 0x7fc2c406d810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40c5170 tx=0x7fc2d4008030 comp rx=0 tx=0).stop 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 msgr2=0x7fc2e4151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 0x7fc2e4151200 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800c7f0 tx=0x7fc2d800cbb0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 shutdown_connections 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc2c406b350 0x7fc2c406d810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc2e40b3370 0x7fc2e40c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc2e415bdd0 0x7fc2e40af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 --2- 172.21.15.77:0/3775988492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40d1c70 0x7fc2e4151200 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 >> 172.21.15.77:0/3775988492 conn(0x7fc2e40731a0 msgr2=0x7fc2e415f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:47.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 shutdown_connections 2024-01-15T14:35:47.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.456+0000 7fc2e8f44700 1 -- 172.21.15.77:0/3775988492 wait complete. 2024-01-15T14:35:47.458 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:47.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.589+0000 7f8a50c4a700 1 Processor -- start 2024-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.589+0000 7f8a50c4a700 1 -- start start 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a50c4a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a4c15e080 0x7f8a4c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a50c4a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0ae430 con 0x7f8a4c06c710 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a50c4a700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0ae710 con 0x7f8a4c0af4d0 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a50c4a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c0af0b0 con 0x7f8a4c15e080 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a4a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a4c15e080 0x7f8a4c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a4a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a4c15e080 0x7f8a4c0b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40662/0 (socket says 172.21.15.77:40662) 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a4ad9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8a4c06c710 0x7f8a4c06e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47366/0 (socket says 172.21.15.77:47366) 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.590+0000 7f8a4a59c700 1 -- 172.21.15.77:0/1448190335 learned_addr learned my addr 172.21.15.77:0/1448190335 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4a59c700 1 -- 172.21.15.77:0/1448190335 >> v1:172.21.15.159:6789/0 conn(0x7f8a4c0af4d0 legacy=0x7f8a4c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4a59c700 1 -- 172.21.15.77:0/1448190335 >> v1:172.21.15.77:6789/0 conn(0x7f8a4c06c710 legacy=0x7f8a4c06e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4a59c700 1 -- 172.21.15.77:0/1448190335 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0ae9f0 con 0x7f8a4c15e080 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4959a700 1 -- 172.21.15.77:0/1448190335 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068528703 0 0) 0x7f8a4c0ae430 con 0x7f8a4c06c710 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4a59c700 1 --2- 172.21.15.77:0/1448190335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c15e080 0x7f8a4c0b5050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0098f0 tx=0x7f8a3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abf0c9e5eb11b2e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4959a700 1 -- 172.21.15.77:0/1448190335 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c004230 con 0x7f8a4c15e080 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4959a700 1 -- 172.21.15.77:0/1448190335 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c004390 con 0x7f8a4c15e080 2024-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a4959a700 1 -- 172.21.15.77:0/1448190335 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0036a0 con 0x7f8a4c15e080 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a50c4a700 1 -- 172.21.15.77:0/1448190335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c15e080 msgr2=0x7f8a4c0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.591+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/1448190335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c15e080 0x7f8a4c0b5050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0098f0 tx=0x7f8a3c009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.592+0000 7f8a50c4a700 1 -- 172.21.15.77:0/1448190335 shutdown_connections 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.592+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/1448190335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c15e080 0x7f8a4c0b5050 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.592+0000 7f8a50c4a700 1 -- 172.21.15.77:0/1448190335 >> 172.21.15.77:0/1448190335 conn(0x7f8a4c071b90 msgr2=0x7f8a4c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.592+0000 7f8a50c4a700 1 -- 172.21.15.77:0/1448190335 shutdown_connections 2024-01-15T14:35:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.592+0000 7f8a50c4a700 1 -- 172.21.15.77:0/1448190335 wait complete. 2024-01-15T14:35:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.593+0000 7f8a50c4a700 1 Processor -- start 2024-01-15T14:35:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.593+0000 7f8a50c4a700 1 -- start start 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 0x7f8a4c160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4c0af4d0 0x7f8a4c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4c15e080 0x7f8a4c158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c15e9a0 con 0x7f8a4c06c710 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c155050 con 0x7f8a4c15e080 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c155330 con 0x7f8a4c0af4d0 2024-01-15T14:35:47.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 0x7f8a4c160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 0x7f8a4c160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40670/0 (socket says 172.21.15.77:40670) 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 -- 172.21.15.77:0/2504389983 learned_addr learned my addr 172.21.15.77:0/2504389983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 -- 172.21.15.77:0/2504389983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4c0af4d0 msgr2=0x7f8a4c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4c0af4d0 0x7f8a4c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 -- 172.21.15.77:0/2504389983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4c15e080 msgr2=0x7f8a4c158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4c15e080 0x7f8a4c158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 -- 172.21.15.77:0/2504389983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c009550 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a4a59c700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 0x7f8a4c160ff0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c000c00 tx=0x7f8a3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00b660 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c00b7e0 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c155610 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00bb90 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.594+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c159080 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.595+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c0753f0 con 0x7f8a4c06c710 2024-01-15T14:35:47.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.596+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8a3c00bcf0 con 0x7f8a4c06c710 2024-01-15T14:35:47.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.596+0000 7f8a3affd700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8a2c06b350 0x7f8a2c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.596+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8a3c08b630 con 0x7f8a4c06c710 2024-01-15T14:35:47.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.596+0000 7f8a49d9b700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8a2c06b350 0x7f8a2c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.597+0000 7f8a49d9b700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8a2c06b350 0x7f8a2c06d810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c06e7f0 tx=0x7f8a40007a40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.601+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a3c0564b0 con 0x7f8a4c06c710 2024-01-15T14:35:47.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.833+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a4c0b0570 con 0x7f8a4c06c710 2024-01-15T14:35:47.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.834+0000 7f8a3affd700 1 -- 172.21.15.77:0/2504389983 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f8a3c059ad0 con 0x7f8a4c06c710 2024-01-15T14:35:47.834 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:47.834 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6834/1904475995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":1904475995},{"type":"v1","addr":"172.21.15.159:6834","nonce":1904475995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2076192929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2076192929},{"type":"v1","addr":"172.21.15.77:6836","nonce":2076192929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3411381208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3411381208},{"type":"v1","addr":"172.21.15.77:6837","nonce":3411381208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:04.589892+0000","modified":"2024-01-15T14:35:07.623076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4278},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4278":{"gid":4278,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.159:6835/1253210024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6833","nonce":1253210024},{"type":"v1","addr":"172.21.15.159:6835","nonce":1253210024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8a2c06b350 msgr2=0x7f8a2c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8a2c06b350 0x7f8a2c06d810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c06e7f0 tx=0x7f8a40007a40 comp rx=0 tx=0).stop 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 msgr2=0x7f8a4c160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 0x7f8a4c160ff0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c000c00 tx=0x7f8a3c004750 comp rx=0 tx=0).stop 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 shutdown_connections 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8a2c06b350 0x7f8a2c06d810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8a4c15e080 0x7f8a4c158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c06c710 0x7f8a4c160ff0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 --2- 172.21.15.77:0/2504389983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8a4c0af4d0 0x7f8a4c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 >> 172.21.15.77:0/2504389983 conn(0x7f8a4c071b90 msgr2=0x7f8a4c0b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 shutdown_connections 2024-01-15T14:35:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.836+0000 7f8a50c4a700 1 -- 172.21.15.77:0/2504389983 wait complete. 2024-01-15T14:35:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-01-15T14:35:47.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:47.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.970+0000 7f5ef78ab700 1 Processor -- start 2024-01-15T14:35:47.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.970+0000 7f5ef78ab700 1 -- start start 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef78ab700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ef006f460 0x7f5ef00b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef78ab700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef00b4ef0 0x7f5ef00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef78ab700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef006bf50 con 0x7f5ef015ac50 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef78ab700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00b5380 con 0x7f5ef006f460 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef78ab700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef015b030 con 0x7f5ef00b4ef0 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ef006f460 0x7f5ef00b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ef006f460 0x7f5ef00b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40674/0 (socket says 172.21.15.77:40674) 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 -- 172.21.15.77:0/137489437 learned_addr learned my addr 172.21.15.77:0/137489437 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef5e48700 1 --2- 172.21.15.77:0/137489437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef00b4ef0 0x7f5ef00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 -- 172.21.15.77:0/137489437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef00b4ef0 msgr2=0x7f5ef00cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 --2- 172.21.15.77:0/137489437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef00b4ef0 0x7f5ef00cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 -- 172.21.15.77:0/137489437 >> v1:172.21.15.159:6789/0 conn(0x7f5ef015ac50 legacy=0x7f5ef006c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 -- 172.21.15.77:0/137489437 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef006e810 con 0x7f5ef006f460 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ef4e46700 1 --2- 172.21.15.77:0/137489437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef00b49b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00098f0 tx=0x7f5ee0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47858e3a04641aba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ee7fff700 1 -- 172.21.15.77:0/137489437 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0004230 con 0x7f5ef006f460 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ee7fff700 1 -- 172.21.15.77:0/137489437 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0004390 con 0x7f5ef006f460 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.971+0000 7f5ee7fff700 1 -- 172.21.15.77:0/137489437 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00036a0 con 0x7f5ef006f460 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 -- 172.21.15.77:0/137489437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 msgr2=0x7f5ef00b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/137489437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef00b49b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00098f0 tx=0x7f5ee0009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 -- 172.21.15.77:0/137489437 shutdown_connections 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/137489437 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef00b4ef0 0x7f5ef00cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/137489437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef00b49b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 -- 172.21.15.77:0/137489437 >> 172.21.15.77:0/137489437 conn(0x7f5ef00b3130 msgr2=0x7f5ef00d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 -- 172.21.15.77:0/137489437 shutdown_connections 2024-01-15T14:35:47.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.972+0000 7f5ef78ab700 1 -- 172.21.15.77:0/137489437 wait complete. 2024-01-15T14:35:47.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.973+0000 7f5ef78ab700 1 Processor -- start 2024-01-15T14:35:47.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.973+0000 7f5ef78ab700 1 -- start start 2024-01-15T14:35:47.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef78ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef01e4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef78ab700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ef00b4ef0 0x7f5ef01558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef78ab700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef015ac50 0x7f5ef0159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef78ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00cb5e0 con 0x7f5ef006f460 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef78ab700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0156840 con 0x7f5ef00b4ef0 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef78ab700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0155df0 con 0x7f5ef015ac50 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef01e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef01e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40684/0 (socket says 172.21.15.77:40684) 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 -- 172.21.15.77:0/3264023830 learned_addr learned my addr 172.21.15.77:0/3264023830 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef4e46700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ef00b4ef0 0x7f5ef01558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5e48700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef015ac50 0x7f5ef0159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 -- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef015ac50 msgr2=0x7f5ef0159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef015ac50 0x7f5ef0159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 -- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ef00b4ef0 msgr2=0x7f5ef01558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ef00b4ef0 0x7f5ef01558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.974+0000 7f5ef5647700 1 -- 172.21.15.77:0/3264023830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0009550 con 0x7f5ef006f460 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.975+0000 7f5ef5647700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef01e4510 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00b4a0 tx=0x7f5eec00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.975+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec011830 con 0x7f5ef006f460 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.975+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec011990 con 0x7f5ef006f460 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.975+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef01e4c10 con 0x7f5ef006f460 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.975+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec011d40 con 0x7f5ef006f460 2024-01-15T14:35:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.975+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef00caf90 con 0x7f5ef006f460 2024-01-15T14:35:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.976+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef0067f60 con 0x7f5ef006f460 2024-01-15T14:35:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.976+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5eec01a400 con 0x7f5ef006f460 2024-01-15T14:35:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.977+0000 7f5ee5ffb700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ed406b350 0x7f5ed406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.977+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5eec089850 con 0x7f5ef006f460 2024-01-15T14:35:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.977+0000 7f5ef4e46700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ed406b350 0x7f5ed406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.977+0000 7f5ef4e46700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ed406b350 0x7f5ed406d810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009130 tx=0x7f5ee0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:47.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:47.981+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5eec054640 con 0x7f5ef006f460 2024-01-15T14:35:48.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.175+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ef006e350 con 0x7f5ef006f460 2024-01-15T14:35:48.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.176+0000 7f5ee5ffb700 1 -- 172.21.15.77:0/3264023830 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16753 (secure 0 0 0) 0x7f5eec015090 con 0x7f5ef006f460 2024-01-15T14:35:48.176 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:48.176 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:46.233118+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:02.432547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:03.553935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:48.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ed406b350 msgr2=0x7f5ed406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ed406b350 0x7f5ed406d810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009130 tx=0x7f5ee0008030 comp rx=0 tx=0).stop 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 msgr2=0x7f5ef01e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef01e4510 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00b4a0 tx=0x7f5eec00b7b0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 shutdown_connections 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ed406b350 0x7f5ed406d810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5ef015ac50 0x7f5ef0159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef006f460 0x7f5ef01e4510 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 --2- 172.21.15.77:0/3264023830 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5ef00b4ef0 0x7f5ef01558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 >> 172.21.15.77:0/3264023830 conn(0x7f5ef00b3130 msgr2=0x7f5ef00cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 shutdown_connections 2024-01-15T14:35:48.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.178+0000 7f5ef78ab700 1 -- 172.21.15.77:0/3264023830 wait complete. 2024-01-15T14:35:48.191 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-15T14:35:48.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.313+0000 7f690c74f700 1 Processor -- start 2024-01-15T14:35:48.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.313+0000 7f690c74f700 1 -- start start 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690c74f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69040ce140 0x7f690415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690c74f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f690415f630 0x7f690406b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690c74f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69041414c0 con 0x7f690415af10 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690c74f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690415c300 con 0x7f69040ce140 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690c74f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040ce9f0 con 0x7f690415f630 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f6909cea700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69040ce140 0x7f690415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690a4eb700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f690415af10 0x7f690415e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47376/0 (socket says 172.21.15.77:47376) 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.314+0000 7f690a4eb700 1 -- 172.21.15.77:0/2224956407 learned_addr learned my addr 172.21.15.77:0/2224956407 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f690acec700 1 --2- 172.21.15.77:0/2224956407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f690415f630 0x7f690406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f6909cea700 1 -- 172.21.15.77:0/2224956407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f690415f630 msgr2=0x7f690406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:48.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f6909cea700 1 --2- 172.21.15.77:0/2224956407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f690415f630 0x7f690406b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f69094e9700 1 -- 172.21.15.77:0/2224956407 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064054320 0 0) 0x7f69041414c0 con 0x7f690415af10 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f6909cea700 1 -- 172.21.15.77:0/2224956407 >> v1:172.21.15.77:6789/0 conn(0x7f690415af10 legacy=0x7f690415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f6909cea700 1 -- 172.21.15.77:0/2224956407 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040cecd0 con 0x7f69040ce140 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f6909cea700 1 --2- 172.21.15.77:0/2224956407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040ce140 0x7f690415f0f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6900009840 tx=0x7f6900009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd20d81fba1a4c90 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f69094e9700 1 -- 172.21.15.77:0/2224956407 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900011830 con 0x7f69040ce140 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f69094e9700 1 -- 172.21.15.77:0/2224956407 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6900011990 con 0x7f69040ce140 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.315+0000 7f69094e9700 1 -- 172.21.15.77:0/2224956407 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900011d20 con 0x7f69040ce140 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 -- 172.21.15.77:0/2224956407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040ce140 msgr2=0x7f690415f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 --2- 172.21.15.77:0/2224956407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040ce140 0x7f690415f0f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6900009840 tx=0x7f6900009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 -- 172.21.15.77:0/2224956407 shutdown_connections 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 --2- 172.21.15.77:0/2224956407 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f690415f630 0x7f690406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 --2- 172.21.15.77:0/2224956407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040ce140 0x7f690415f0f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 -- 172.21.15.77:0/2224956407 >> 172.21.15.77:0/2224956407 conn(0x7f69040cd570 msgr2=0x7f69040cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 -- 172.21.15.77:0/2224956407 shutdown_connections 2024-01-15T14:35:48.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.316+0000 7f690c74f700 1 -- 172.21.15.77:0/2224956407 wait complete. 2024-01-15T14:35:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.317+0000 7f690c74f700 1 Processor -- start 2024-01-15T14:35:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.317+0000 7f690c74f700 1 -- start start 2024-01-15T14:35:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69040ce140 0x7f690413c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f690415af10 0x7f6904156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 0x7f690415df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690415a5d0 con 0x7f69040b1920 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904159ce0 con 0x7f690415af10 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040b2070 con 0x7f69040ce140 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 0x7f690415df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 0x7f690415df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40708/0 (socket says 172.21.15.77:40708) 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 -- 172.21.15.77:0/3251982854 learned_addr learned my addr 172.21.15.77:0/3251982854 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690a4eb700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69040ce140 0x7f690413c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f6909cea700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f690415af10 0x7f6904156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 -- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69040ce140 msgr2=0x7f690413c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69040ce140 0x7f690413c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 -- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f690415af10 msgr2=0x7f6904156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f690415af10 0x7f6904156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 -- 172.21.15.77:0/3251982854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000094a0 con 0x7f69040b1920 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690acec700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 0x7f690415df40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00cc30 tx=0x7f68fc00a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00ab70 con 0x7f69040b1920 2024-01-15T14:35:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00acd0 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69040b1db0 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc003c40 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.318+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69040b25a0 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.319+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690415dc60 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.320+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f68fc00f3f0 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.320+0000 7f68f6ffd700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68e806b350 0x7f68e806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.320+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f68fc089e60 con 0x7f69040b1920 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.320+0000 7f690a4eb700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68e806b350 0x7f68e806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.320+0000 7f690a4eb700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68e806b350 0x7f68e806d810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f68f8000f60 tx=0x7f68f8007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.325+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f690415dc60 con 0x7f69040b1920 2024-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.556+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f690415dc60 con 0x7f69040b1920 2024-01-15T14:35:48.557 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-01-15T14:35:48.557+0000 7fe68c60c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.560+0000 7f68f6ffd700 1 -- 172.21.15.77:0/3251982854 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f690415dc60 con 0x7f69040b1920 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.562+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68e806b350 msgr2=0x7f68e806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.562+0000 7f690c74f700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68e806b350 0x7f68e806d810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f68f8000f60 tx=0x7f68f8007c00 comp rx=0 tx=0).stop 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.562+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 msgr2=0x7f690415df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.562+0000 7f690c74f700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 0x7f690415df40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00cc30 tx=0x7f68fc00a1f0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 shutdown_connections 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68e806b350 0x7f68e806d810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f69040ce140 0x7f690413c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040b1920 0x7f690415df40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 --2- 172.21.15.77:0/3251982854 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f690415af10 0x7f6904156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 >> 172.21.15.77:0/3251982854 conn(0x7f69040cd570 msgr2=0x7f690415b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 shutdown_connections 2024-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.563+0000 7f690c74f700 1 -- 172.21.15.77:0/3251982854 wait complete. 2024-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-15T14:35:48.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-15T14:35:48.602 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-01-15T14:35:48.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.698+0000 7f9f79841700 1 Processor -- start 2024-01-15T14:35:48.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.698+0000 7f9f79841700 1 -- start start 2024-01-15T14:35:48.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.699+0000 7f9f79841700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9f7415f8d0 0x7f9f740b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.699+0000 7f9f79841700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f740d03a0 0x7f9f740ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.699+0000 7f9f79841700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f740b3860 con 0x7f9f740cffc0 2024-01-15T14:35:48.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.699+0000 7f9f79841700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f740b2520 con 0x7f9f7415f8d0 2024-01-15T14:35:48.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.699+0000 7f9f79841700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f740b2800 con 0x7f9f740d03a0 2024-01-15T14:35:48.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f727fc700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f9f7415f8d0 0x7f9f740b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f737fe700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f740d03a0 0x7f9f740ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f72ffd700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9f740cffc0 0x7f9f7415f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:52562/0 (socket says 172.21.15.77:52562) 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f72ffd700 1 -- 172.21.15.77:0/49043120 learned_addr learned my addr 172.21.15.77:0/49043120 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f71ffb700 1 -- 172.21.15.77:0/49043120 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767229153 0 0) 0x7f9f68001110 con 0x7f9f740cffc0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f71ffb700 1 -- 172.21.15.77:0/49043120 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5c0035b0 con 0x7f9f740cffc0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f727fc700 1 -- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f740d03a0 msgr2=0x7f9f740ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f727fc700 1 --2- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f740d03a0 0x7f9f740ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f727fc700 1 -- 172.21.15.77:0/49043120 >> v1:172.21.15.159:6789/0 conn(0x7f9f740cffc0 legacy=0x7f9f7415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f727fc700 1 -- 172.21.15.77:0/49043120 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740b2ae0 con 0x7f9f7415f8d0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f727fc700 1 --2- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f7415f8d0 0x7f9f740b3320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009840 tx=0x7f9f64009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8de0f1699e912212 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f71ffb700 1 -- 172.21.15.77:0/49043120 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64011830 con 0x7f9f7415f8d0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f71ffb700 1 -- 172.21.15.77:0/49043120 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64011990 con 0x7f9f7415f8d0 2024-01-15T14:35:48.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.700+0000 7f9f71ffb700 1 -- 172.21.15.77:0/49043120 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64011d00 con 0x7f9f7415f8d0 2024-01-15T14:35:48.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 -- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f7415f8d0 msgr2=0x7f9f740b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 --2- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f7415f8d0 0x7f9f740b3320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009840 tx=0x7f9f64009b50 comp rx=0 tx=0).stop 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 -- 172.21.15.77:0/49043120 shutdown_connections 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 --2- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f740d03a0 0x7f9f740ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 --2- 172.21.15.77:0/49043120 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f7415f8d0 0x7f9f740b3320 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 -- 172.21.15.77:0/49043120 >> 172.21.15.77:0/49043120 conn(0x7f9f740d1e50 msgr2=0x7f9f740d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 -- 172.21.15.77:0/49043120 shutdown_connections 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.701+0000 7f9f79841700 1 -- 172.21.15.77:0/49043120 wait complete. 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.702+0000 7f9f79841700 1 Processor -- start 2024-01-15T14:35:48.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.702+0000 7f9f79841700 1 -- start start 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f79841700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 0x7f9f740cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f79841700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f740d03a0 0x7f9f741f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f79841700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f7415f8d0 0x7f9f740b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f79841700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f740c9250 con 0x7f9f740cffc0 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f79841700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f741f3750 con 0x7f9f740d03a0 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f79841700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f740c8f20 con 0x7f9f7415f8d0 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 0x7f9f740cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 0x7f9f740cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40710/0 (socket says 172.21.15.77:40710) 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 -- 172.21.15.77:0/1430611718 learned_addr learned my addr 172.21.15.77:0/1430611718 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f727fc700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f740d03a0 0x7f9f741f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f737fe700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f7415f8d0 0x7f9f740b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 -- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f7415f8d0 msgr2=0x7f9f740b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f7415f8d0 0x7f9f740b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 -- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f740d03a0 msgr2=0x7f9f741f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f740d03a0 0x7f9f741f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.703+0000 7f9f72ffd700 1 -- 172.21.15.77:0/1430611718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f640094a0 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.704+0000 7f9f72ffd700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 0x7f9f740cb980 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800c4d0 tx=0x7f9f6800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.704+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68005d70 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.704+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f680070d0 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.704+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68007480 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.704+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f740b7390 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.704+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f740b7950 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.705+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f740c57f0 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.705+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9f680075e0 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.705+0000 7f9f5bfff700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9f6006b350 0x7f9f6006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.705+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f9f68088e50 con 0x7f9f740cffc0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.705+0000 7f9f727fc700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9f6006b350 0x7f9f6006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:48.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.706+0000 7f9f727fc700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9f6006b350 0x7f9f6006d810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009fd0 tx=0x7f9f64008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:48.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.710+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f68053c80 con 0x7f9f740cffc0 2024-01-15T14:35:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:48.947+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9f740b3860 con 0x7f9f740cffc0 2024-01-15T14:35:49.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.564+0000 7f9f5bfff700 1 -- 172.21.15.77:0/1430611718 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9f740b3860 con 0x7f9f740cffc0 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9f6006b350 msgr2=0x7f9f6006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9f6006b350 0x7f9f6006d810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009fd0 tx=0x7f9f64008030 comp rx=0 tx=0).stop 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 msgr2=0x7f9f740cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 0x7f9f740cb980 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800c4d0 tx=0x7f9f6800c7e0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 shutdown_connections 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9f6006b350 0x7f9f6006d810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9f7415f8d0 0x7f9f740b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f740cffc0 0x7f9f740cb980 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 --2- 172.21.15.77:0/1430611718 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9f740d03a0 0x7f9f741f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 >> 172.21.15.77:0/1430611718 conn(0x7f9f740d1e50 msgr2=0x7f9f740b08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:49.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 shutdown_connections 2024-01-15T14:35:49.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.567+0000 7f9f79841700 1 -- 172.21.15.77:0/1430611718 wait complete. 2024-01-15T14:35:49.582 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-15T14:35:49.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.710+0000 7f4310ca1700 1 Processor -- start 2024-01-15T14:35:49.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.710+0000 7f4310ca1700 1 -- start start 2024-01-15T14:35:49.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f4310ca1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c1622b0 0x7f430c06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:49.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f4310ca1700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f430c162820 0x7f430c06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f4310ca1700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c06bed0 con 0x7f430c15df20 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f4310ca1700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430c06cfe0 con 0x7f430c162820 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f4310ca1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c06bba0 con 0x7f430c1622b0 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f430c162820 0x7f430c06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f430c162820 0x7f430c06caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55024/0 (socket says 172.21.15.77:55024) 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 -- 172.21.15.77:0/3407726880 learned_addr learned my addr 172.21.15.77:0/3407726880 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f4309d9b700 1 --2- 172.21.15.77:0/3407726880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c1622b0 0x7f430c06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 -- 172.21.15.77:0/3407726880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c1622b0 msgr2=0x7f430c06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 --2- 172.21.15.77:0/3407726880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c1622b0 0x7f430c06c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 -- 172.21.15.77:0/3407726880 >> v1:172.21.15.159:6789/0 conn(0x7f430c15df20 legacy=0x7f430c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 -- 172.21.15.77:0/3407726880 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c070060 con 0x7f430c162820 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430ad9d700 1 --2- 172.21.15.77:0/3407726880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c162820 0x7f430c06caa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4304009e50 tx=0x7f430400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23270f7f268715 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430959a700 1 -- 172.21.15.77:0/3407726880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430400e070 con 0x7f430c162820 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430959a700 1 -- 172.21.15.77:0/3407726880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304003a20 con 0x7f430c162820 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.711+0000 7f430959a700 1 -- 172.21.15.77:0/3407726880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304003dd0 con 0x7f430c162820 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 -- 172.21.15.77:0/3407726880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c162820 msgr2=0x7f430c06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 --2- 172.21.15.77:0/3407726880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c162820 0x7f430c06caa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4304009e50 tx=0x7f430400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 -- 172.21.15.77:0/3407726880 shutdown_connections 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 --2- 172.21.15.77:0/3407726880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c162820 0x7f430c06caa0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 --2- 172.21.15.77:0/3407726880 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c1622b0 0x7f430c06c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 -- 172.21.15.77:0/3407726880 >> 172.21.15.77:0/3407726880 conn(0x7f430c06dc80 msgr2=0x7f430c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 -- 172.21.15.77:0/3407726880 shutdown_connections 2024-01-15T14:35:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.712+0000 7f4310ca1700 1 -- 172.21.15.77:0/3407726880 wait complete. 2024-01-15T14:35:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.713+0000 7f4310ca1700 1 Processor -- start 2024-01-15T14:35:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.713+0000 7f4310ca1700 1 -- start start 2024-01-15T14:35:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4310ca1700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f430c15df20 0x7f430c0ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4310ca1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 0x7f430c0b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4310ca1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c162820 0x7f430c156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4310ca1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0c92a0 con 0x7f430c1622b0 2024-01-15T14:35:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4310ca1700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0c9580 con 0x7f430c15df20 2024-01-15T14:35:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4310ca1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0c9860 con 0x7f430c162820 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4309d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 0x7f430c0b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4309d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 0x7f430c0b70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55040/0 (socket says 172.21.15.77:55040) 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f4309d9b700 1 -- 172.21.15.77:0/2298307339 learned_addr learned my addr 172.21.15.77:0/2298307339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f430ad9d700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c162820 0x7f430c156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.714+0000 7f430a59c700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f430c15df20 0x7f430c0ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4309d9b700 1 -- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c162820 msgr2=0x7f430c156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4309d9b700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c162820 0x7f430c156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4309d9b700 1 -- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f430c15df20 msgr2=0x7f430c0ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4309d9b700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f430c15df20 0x7f430c0ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4309d9b700 1 -- 172.21.15.77:0/2298307339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304009b30 con 0x7f430c1622b0 2024-01-15T14:35:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4309d9b700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 0x7f430c0b70f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f430000d800 tx=0x7f430000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43000125f0 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300012750 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300012b00 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c0c9b40 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.715+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c0c5940 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.716+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c0751e0 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.716+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4300012c60 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.717+0000 7f42faffd700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f42ec06b350 0x7f42ec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.717+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f4300089ca0 con 0x7f430c1622b0 2024-01-15T14:35:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.717+0000 7f430a59c700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f42ec06b350 0x7f42ec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.717+0000 7f430a59c700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f42ec06b350 0x7f42ec06d810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009960 tx=0x7f42fc0090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.721+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4300054af0 con 0x7f430c1622b0 2024-01-15T14:35:49.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:49.913+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f430c067f60 con 0x7f430c1622b0 2024-01-15T14:35:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.567+0000 7f42faffd700 1 -- 172.21.15.77:0/2298307339 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4300058110 con 0x7f430c1622b0 2024-01-15T14:35:50.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f42ec06b350 msgr2=0x7f42ec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:50.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f42ec06b350 0x7f42ec06d810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009960 tx=0x7f42fc0090e0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 msgr2=0x7f430c0b70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 0x7f430c0b70f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f430000d800 tx=0x7f430000db10 comp rx=0 tx=0).stop 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 shutdown_connections 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f42ec06b350 0x7f42ec06d810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f430c162820 0x7f430c156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f430c15df20 0x7f430c0ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 --2- 172.21.15.77:0/2298307339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f430c1622b0 0x7f430c0b70f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 >> 172.21.15.77:0/2298307339 conn(0x7f430c06dc80 msgr2=0x7f430c0aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 shutdown_connections 2024-01-15T14:35:50.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.571+0000 7f4310ca1700 1 -- 172.21.15.77:0/2298307339 wait complete. 2024-01-15T14:35:50.574 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-01-15T14:35:50.587 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-15T14:35:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.718+0000 7f4f6ae53700 1 Processor -- start 2024-01-15T14:35:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.718+0000 7f4f6ae53700 1 -- start start 2024-01-15T14:35:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f6ae53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64159b70 0x7f4f640d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f6ae53700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f64072540 0x7f4f64074410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f6ae53700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f64159f50 con 0x7f4f64073f90 2024-01-15T14:35:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f6ae53700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f640729d0 con 0x7f4f64072540 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f6ae53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64074950 con 0x7f4f64159b70 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f693f0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f64072540 0x7f4f64074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f63fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4f64073f90 0x7f4f64071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45356/0 (socket says 172.21.15.77:45356) 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f63fff700 1 -- 172.21.15.77:0/4252633064 learned_addr learned my addr 172.21.15.77:0/4252633064 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f693f0700 1 -- 172.21.15.77:0/4252633064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64159b70 msgr2=0x7f4f640d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f68bef700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64159b70 0x7f4f640d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f693f0700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64159b70 0x7f4f640d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f637fe700 1 -- 172.21.15.77:0/4252633064 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906467452 0 0) 0x7f4f64159f50 con 0x7f4f64073f90 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f693f0700 1 -- 172.21.15.77:0/4252633064 >> v1:172.21.15.77:6789/0 conn(0x7f4f64073f90 legacy=0x7f4f64071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f693f0700 1 -- 172.21.15.77:0/4252633064 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640cf730 con 0x7f4f64072540 2024-01-15T14:35:50.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f68bef700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64159b70 0x7f4f640d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.719+0000 7f4f693f0700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64072540 0x7f4f64074410 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009dc0 tx=0x7f4f5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3290e54743369456 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f637fe700 1 -- 172.21.15.77:0/4252633064 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00e070 con 0x7f4f64072540 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f637fe700 1 -- 172.21.15.77:0/4252633064 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c00bb90 con 0x7f4f64072540 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f637fe700 1 -- 172.21.15.77:0/4252633064 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c012430 con 0x7f4f64072540 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 -- 172.21.15.77:0/4252633064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64072540 msgr2=0x7f4f64074410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64072540 0x7f4f64074410 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009dc0 tx=0x7f4f5c00b330 comp rx=0 tx=0).stop 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 -- 172.21.15.77:0/4252633064 shutdown_connections 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64072540 0x7f4f64074410 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/4252633064 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64159b70 0x7f4f640d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 -- 172.21.15.77:0/4252633064 >> 172.21.15.77:0/4252633064 conn(0x7f4f640d0af0 msgr2=0x7f4f640d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:50.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 -- 172.21.15.77:0/4252633064 shutdown_connections 2024-01-15T14:35:50.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.720+0000 7f4f6ae53700 1 -- 172.21.15.77:0/4252633064 wait complete. 2024-01-15T14:35:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.721+0000 7f4f6ae53700 1 Processor -- start 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 -- start start 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f64072540 0x7f4f640cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64073f90 0x7f4f641e3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 0x7f4f64143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641e4050 con 0x7f4f64159b70 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641e4330 con 0x7f4f64072540 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f6ae53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641e4610 con 0x7f4f64073f90 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.722+0000 7f4f693f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 0x7f4f64143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 0x7f4f64143080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55052/0 (socket says 172.21.15.77:55052) 2024-01-15T14:35:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 -- 172.21.15.77:0/454695525 learned_addr learned my addr 172.21.15.77:0/454695525 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f63fff700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64073f90 0x7f4f641e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f68bef700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f64072540 0x7f4f640cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 -- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64073f90 msgr2=0x7f4f641e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64073f90 0x7f4f641e3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 -- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f64072540 msgr2=0x7f4f640cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f64072540 0x7f4f640cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 -- 172.21.15.77:0/454695525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c009a70 con 0x7f4f64159b70 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f693f0700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 0x7f4f64143080 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c014040 tx=0x7f4f5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c01b810 con 0x7f4f64159b70 2024-01-15T14:35:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c020070 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c01bbd0 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f641e48f0 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.723+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f641e50f0 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.724+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f641e4e10 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.725+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4f5c01bd30 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.725+0000 7f4f617fa700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4f4806b350 0x7f4f4806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.725+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f4f5c022070 con 0x7f4f64159b70 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.725+0000 7f4f68bef700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4f4806b350 0x7f4f4806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.725+0000 7f4f68bef700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4f4806b350 0x7f4f4806d810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006fd0 tx=0x7f4f58008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.729+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f641e4e10 con 0x7f4f64159b70 2024-01-15T14:35:50.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:50.922+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4f641e4e10 con 0x7f4f64159b70 2024-01-15T14:35:51.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.571+0000 7f4f617fa700 1 -- 172.21.15.77:0/454695525 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4f641e4e10 con 0x7f4f64159b70 2024-01-15T14:35:51.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4f4806b350 msgr2=0x7f4f4806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4f4806b350 0x7f4f4806d810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006fd0 tx=0x7f4f58008040 comp rx=0 tx=0).stop 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 msgr2=0x7f4f64143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 0x7f4f64143080 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c014040 tx=0x7f4f5c004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 shutdown_connections 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4f4806b350 0x7f4f4806d810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f64159b70 0x7f4f64143080 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4f64072540 0x7f4f640cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 --2- 172.21.15.77:0/454695525 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4f64073f90 0x7f4f641e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.574+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 >> 172.21.15.77:0/454695525 conn(0x7f4f640d0af0 msgr2=0x7f4f6415a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:51.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.575+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 shutdown_connections 2024-01-15T14:35:51.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.575+0000 7f4f6ae53700 1 -- 172.21.15.77:0/454695525 wait complete. 2024-01-15T14:35:51.577 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-01-15T14:35:51.591 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T14:35:51.592 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T14:35:51.592 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T14:35:51.592 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:35:51.592 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T14:35:51.593 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T14:35:51.593 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T14:35:51.593 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:35:51.593 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T14:35:51.593 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T14:35:51.594 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T14:35:51.594 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:35:51.594 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T14:35:51.594 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T14:35:51.594 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T14:35:51.595 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T14:35:51.603 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T14:35:51.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T14:35:51.605 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T14:35:51.605 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T14:35:51.608 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T14:35:51.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T14:35:51.609 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T14:35:51.610 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T14:35:51.647 INFO:tasks.ceph.mds.a:Started 2024-01-15T14:35:51.647 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:51.648 INFO:tasks.ceph.mds.c:Started 2024-01-15T14:35:51.648 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:51.651 INFO:tasks.ceph.mds.b:Started 2024-01-15T14:35:51.651 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:51.652 INFO:tasks.ceph.mds.d:Started 2024-01-15T14:35:51.652 DEBUG:teuthology.parallel:result is None 2024-01-15T14:35:51.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T14:35:51.732 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-01-15T14:35:51.732 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-01-15T14:35:51.736 INFO:tasks.ceph.mds.d.smithi159.stdout:starting mds.d at 2024-01-15T14:35:51.737 INFO:tasks.ceph.mds.b.smithi159.stdout:starting mds.b at 2024-01-15T14:35:51.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f755df53700 1 Processor -- start 2024-01-15T14:35:51.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f755df53700 1 -- start start 2024-01-15T14:35:51.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f755df53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 0x7f75580ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f755df53700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75580cf530 con 0x7f75580cf0d0 2024-01-15T14:35:51.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f755df53700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755806bc00 con 0x7f755815e0f0 2024-01-15T14:35:51.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f755df53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755815e4d0 con 0x7f75580b2b60 2024-01-15T14:35:51.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f7557fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f75580cf0d0 0x7f75580d1f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45380/0 (socket says 172.21.15.77:45380) 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.787+0000 7f7557fff700 1 -- 172.21.15.77:0/4088812025 learned_addr learned my addr 172.21.15.77:0/4088812025 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7556ffd700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 0x7f75580ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440178459 0 0) 0x7f75580cf530 con 0x7f75580cf0d0 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75440035b0 con 0x7f75580cf0d0 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 667702488 0 0) 0x7f75440035b0 con 0x7f75580cf0d0 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 msgr2=0x7f75580ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 0x7f75580ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 >> v1:172.21.15.159:6789/0 conn(0x7f755815e0f0 legacy=0x7f75580b9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755806aff0 con 0x7f75580cf0d0 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f75500019f0 con 0x7f75580cf0d0 2024-01-15T14:35:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 >> v1:172.21.15.77:6789/0 conn(0x7f75580cf0d0 legacy=0x7f75580d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544003a30 0x7f7544003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 0x7f7544004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7544004d90 0x7f7544005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755815e4d0 con 0x7f7544004d90 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754400bac0 con 0x7f7544003a30 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754400bc80 con 0x7f75580b2b60 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7557fff700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7544004d90 0x7f7544005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f75577fe700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544003a30 0x7f7544003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7556ffd700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 0x7f7544004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7557fff700 1 -- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 msgr2=0x7f7544004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7557fff700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f75580b2b60 0x7f7544004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7557fff700 1 -- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544003a30 msgr2=0x7f7544003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7557fff700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544003a30 0x7f7544003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.788+0000 7f7557fff700 1 -- 172.21.15.77:0/4088812025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440035b0 con 0x7f7544004d90 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f7557fff700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7544004d90 0x7f7544005210 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f755000a800 tx=0x7f7544005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b6842e9669bfbb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755000a2f0 con 0x7f7544004d90 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7550012600 con 0x7f7544004d90 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f75567fc700 1 -- 172.21.15.77:0/4088812025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75500129b0 con 0x7f7544004d90 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f755df53700 1 -- 172.21.15.77:0/4088812025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7544004d90 msgr2=0x7f7544005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f755df53700 1 --2- 172.21.15.77:0/4088812025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7544004d90 0x7f7544005210 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f755000a800 tx=0x7f7544005880 comp rx=0 tx=0).stop 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f7557fff700 1 -- 172.21.15.77:0/4088812025 reap_dead start 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f755df53700 1 -- 172.21.15.77:0/4088812025 shutdown_connections 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f755df53700 1 -- 172.21.15.77:0/4088812025 >> 172.21.15.77:0/4088812025 conn(0x7f75580b5ce0 msgr2=0x7f75580b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f755df53700 1 -- 172.21.15.77:0/4088812025 shutdown_connections 2024-01-15T14:35:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.789+0000 7f755df53700 1 -- 172.21.15.77:0/4088812025 wait complete. 2024-01-15T14:35:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.791+0000 7f755df53700 1 Processor -- start 2024-01-15T14:35:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.791+0000 7f755df53700 1 -- start start 2024-01-15T14:35:51.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f755df53700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544004d90 0x7f75581522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f755df53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 0x7f7558160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f755df53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f755815e610 0x7f75580b83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f755df53700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7558152830 con 0x7f75580ca5d0 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f755df53700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755815ef80 con 0x7f7544004d90 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f755df53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755815f260 con 0x7f755815e610 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 0x7f7558160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 0x7f7558160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55102/0 (socket says 172.21.15.77:55102) 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 -- 172.21.15.77:0/977891253 learned_addr learned my addr 172.21.15.77:0/977891253 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7557fff700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f755815e610 0x7f75580b83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f75577fe700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544004d90 0x7f75581522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 -- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f755815e610 msgr2=0x7f75580b83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f755815e610 0x7f75580b83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 -- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544004d90 msgr2=0x7f75581522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544004d90 0x7f75581522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.792+0000 7f7556ffd700 1 -- 172.21.15.77:0/977891253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755815f540 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.793+0000 7f7556ffd700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 0x7f7558160020 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7548011aa0 tx=0x7f7548011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.793+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800f980 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.793+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754800fae0 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.793+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755815f820 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.793+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800fe90 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.793+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7558160560 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.794+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75580751e0 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.794+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7548014440 con 0x7f75580ca5d0 2024-01-15T14:35:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.794+0000 7f754ffff700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f753806b350 0x7f753806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.795+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7548089fd0 con 0x7f75580ca5d0 2024-01-15T14:35:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.795+0000 7f75577fe700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f753806b350 0x7f753806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.795+0000 7f75577fe700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f753806b350 0x7f753806d810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7540007fd0 tx=0x7f7540009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:51.799+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7548055290 con 0x7f75580ca5d0 2024-01-15T14:35:52.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.008+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f75580ce330 con 0x7f75580ca5d0 2024-01-15T14:35:52.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.580+0000 7f754ffff700 1 -- 172.21.15.77:0/977891253 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f75480588b0 con 0x7f75580ca5d0 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f753806b350 msgr2=0x7f753806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f753806b350 0x7f753806d810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7540007fd0 tx=0x7f7540009040 comp rx=0 tx=0).stop 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 msgr2=0x7f7558160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 0x7f7558160020 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7548011aa0 tx=0x7f7548011e60 comp rx=0 tx=0).stop 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 shutdown_connections 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f753806b350 0x7f753806d810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f755815e610 0x7f75580b83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7544004d90 0x7f75581522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 --2- 172.21.15.77:0/977891253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75580ca5d0 0x7f7558160020 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.583+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 >> 172.21.15.77:0/977891253 conn(0x7f75580b5ce0 msgr2=0x7f75580b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:52.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.584+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 shutdown_connections 2024-01-15T14:35:52.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.584+0000 7f755df53700 1 -- 172.21.15.77:0/977891253 wait complete. 2024-01-15T14:35:52.585 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-01-15T14:35:52.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T14:35:52.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.720+0000 7f994ea53700 1 Processor -- start 2024-01-15T14:35:52.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.720+0000 7f994ea53700 1 -- start start 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f994ea53700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994815e280 0x7f99480cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f994ea53700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f99480cf250 0x7f994806b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f994ea53700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480d0a80 con 0x7f994815dea0 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f994ea53700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99480d2670 con 0x7f994815e280 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f994ea53700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994806c4a0 con 0x7f99480cf250 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f99477fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994815e280 0x7f99480cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f9947fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f994815dea0 0x7f99480d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45386/0 (socket says 172.21.15.77:45386) 2024-01-15T14:35:52.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f9947fff700 1 -- 172.21.15.77:0/1927647592 learned_addr learned my addr 172.21.15.77:0/1927647592 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f99477fe700 1 -- 172.21.15.77:0/1927647592 >> v2:172.21.15.159:3300/0 conn(0x7f99480cf250 msgr2=0x7f994806b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f99477fe700 1 --2- 172.21.15.77:0/1927647592 >> v2:172.21.15.159:3300/0 conn(0x7f99480cf250 0x7f994806b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f99477fe700 1 -- 172.21.15.77:0/1927647592 >> v1:172.21.15.77:6789/0 conn(0x7f994815dea0 legacy=0x7f99480d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f99477fe700 1 -- 172.21.15.77:0/1927647592 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480cfeb0 con 0x7f994815e280 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.721+0000 7f9946ffd700 1 -- 172.21.15.77:0/1927647592 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413350457 0 0) 0x7f99480d0a80 con 0x7f994815dea0 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f99477fe700 1 --2- 172.21.15.77:0/1927647592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815e280 0x7f99480cc2c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f99480d2cd0 tx=0x7f9938009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15724c28300314dc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f9946ffd700 1 -- 172.21.15.77:0/1927647592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938011830 con 0x7f994815e280 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f9946ffd700 1 -- 172.21.15.77:0/1927647592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938011990 con 0x7f994815e280 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f9946ffd700 1 -- 172.21.15.77:0/1927647592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938011d20 con 0x7f994815e280 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 -- 172.21.15.77:0/1927647592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815e280 msgr2=0x7f99480cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 --2- 172.21.15.77:0/1927647592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815e280 0x7f99480cc2c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f99480d2cd0 tx=0x7f9938009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 -- 172.21.15.77:0/1927647592 shutdown_connections 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 --2- 172.21.15.77:0/1927647592 >> v2:172.21.15.159:3300/0 conn(0x7f99480cf250 0x7f994806b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 --2- 172.21.15.77:0/1927647592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815e280 0x7f99480cc2c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 -- 172.21.15.77:0/1927647592 >> 172.21.15.77:0/1927647592 conn(0x7f9948073130 msgr2=0x7f994809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 -- 172.21.15.77:0/1927647592 shutdown_connections 2024-01-15T14:35:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.722+0000 7f994ea53700 1 -- 172.21.15.77:0/1927647592 wait complete. 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.723+0000 7f994ea53700 1 Processor -- start 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 -- start start 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99480cf250 0x7f99480c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 0x7f99480b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9948157f40 0x7f99480c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480b3150 con 0x7f994815dea0 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480b3430 con 0x7f99480cf250 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f994ea53700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480c66a0 con 0x7f9948157f40 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.724+0000 7f99477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 0x7f99480b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 0x7f99480b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55130/0 (socket says 172.21.15.77:55130) 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 -- 172.21.15.77:0/1212950288 learned_addr learned my addr 172.21.15.77:0/1212950288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f9947fff700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99480cf250 0x7f99480c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f994cff0700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9948157f40 0x7f99480c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 -- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9948157f40 msgr2=0x7f99480c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9948157f40 0x7f99480c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 -- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99480cf250 msgr2=0x7f99480c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99480cf250 0x7f99480c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 -- 172.21.15.77:0/1212950288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99380094a0 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f9947fff700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99480cf250 0x7f99480c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f99477fe700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 0x7f99480b2920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9938014040 tx=0x7f9938004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99380038a0 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993801e070 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99480c6980 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938003c60 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.725+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99480c62e0 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.726+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948067f60 con 0x7f994815dea0 2024-01-15T14:35:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.727+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f993802d450 con 0x7f994815dea0 2024-01-15T14:35:52.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.727+0000 7f9944ff9700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f993406b350 0x7f993406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:52.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.727+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f993808a9a0 con 0x7f994815dea0 2024-01-15T14:35:52.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.727+0000 7f9947fff700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f993406b350 0x7f993406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:52.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.727+0000 7f9947fff700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f993406b350 0x7f993406d810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f993c000f60 tx=0x7f993c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:52.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.732+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9938055b60 con 0x7f994815dea0 2024-01-15T14:35:52.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.997+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f99480b7bf0 con 0x7f994815dea0 2024-01-15T14:35:52.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.997+0000 7f9944ff9700 1 -- 172.21.15.77:0/1212950288 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f9938015020 con 0x7f994815dea0 2024-01-15T14:35:52.997 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVljxYOJBAA6tNJno9y9x+zGr1FiZN2hA==", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVl+5FzMBAAra8qc3O5AH9FOYwASZIuag==", 2024-01-15T14:35:52.998 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVlGw9oKxAArSX/vvELUO3UU9nSNCmURg==", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:52.999 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVl7m/ZNxAA41CCPNutEM+PKAaLqzBYng==", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBhQqVlkqHPFRAAJ/qFoPn4R8+VN6fBlM5kFg==", 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.000 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBkQqVlWohgFBAAm6nn3wcmSfWKALwZ63ZR/g==", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-01-15T14:35:53.001 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBnQqVlIvlKExAAXXy2r2cr61BvOivASiwypQ==", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBqQqVlfiVlEhAAXi3wsvTmPRGeBGx7e6pD2w==", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.002 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBwQqVli8LcFhAARSvQ90Fmunp2XMM2r4/BBQ==", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBzQqVlnfmBFRAAUEgbCE4QFeX4xrgIf0rLng==", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.003 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2QqVlOi6XExAA/J69KO/nyj6ggyGu6Tnx3Q==", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5QqVlMFldEhAA7mY2tYGH8cvWIBxqaNqBuw==", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-01-15T14:35:53.004 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBeQqVl3gmtABAA2JDAPlTu6jT/iTUZPZjfVw==", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-01-15T14:35:53.005 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBeQqVlrWl0BxAAQNlFMK3OnPmkHN0sasy5Kw==", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBeQqVlfSMMBBAACXaauvbUylsDhE6gLtK4jw==", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-01-15T14:35:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBeQqVlcqDeChAAe/SEZKUY/KknjkYmcbuj+w==", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVlOQxDBRAAOka6+T1zXjwQzDahWGmVng==", 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.007 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mds", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/QqVl5vkWEhAAqbHXbuesiCrhYjdz6mijYg==", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mds" 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mgr", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/QqVl4wMXEhAAM5A3qwiTe5xJ8nvTVWx6gg==", 2024-01-15T14:35:53.008 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-osd", 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/QqVlUAsXEhAAc7bhapcNlUEa11yCY5hxdA==", 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-osd" 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd", 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/QqVlfhIXEhAAebkB9F63YbH/fUf3H0uYCQ==", 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-15T14:35:53.009 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/QqVlrxsXEhAAMlY8QQlQbVxncoPaz4hTdg==", 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rgw", 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/QqVlfiMXEhAA3ueyptuXOLo2m25YC7gwAw==", 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.010 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVlHpecIBAAxACo580Z7g9U4IHsV17hXQ==", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdQqVlY0v0GhAAua4VooMS6v1Lii5ufEofbg==", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.011 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:53.012 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-01-15T14:35:53.012 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.999+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f993406b350 msgr2=0x7f993406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.999+0000 7f994ea53700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f993406b350 0x7f993406d810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f993c000f60 tx=0x7f993c009040 comp rx=0 tx=0).stop 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.999+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 msgr2=0x7f99480b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:52.999+0000 7f994ea53700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 0x7f99480b2920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9938014040 tx=0x7f9938004750 comp rx=0 tx=0).stop 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 shutdown_connections 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f993406b350 0x7f993406d810 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9948157f40 0x7f99480c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f99480cf250 0x7f99480c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 --2- 172.21.15.77:0/1212950288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994815dea0 0x7f99480b2920 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 >> 172.21.15.77:0/1212950288 conn(0x7f9948073130 msgr2=0x7f994806ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:53.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 shutdown_connections 2024-01-15T14:35:53.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.000+0000 7f994ea53700 1 -- 172.21.15.77:0/1212950288 wait complete. 2024-01-15T14:35:53.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-15T14:35:53.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.136+0000 7eff85a68700 1 Processor -- start 2024-01-15T14:35:53.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.136+0000 7eff85a68700 1 -- start start 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff85a68700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7eff80140f20 0x7eff800b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff85a68700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 0x7eff800b2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff85a68700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff80161b40 con 0x7eff800b2590 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff85a68700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff800b2a20 con 0x7eff80140f20 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff85a68700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff800b3e10 con 0x7eff800b3900 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7eff800b2590 0x7eff80071360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45392/0 (socket says 172.21.15.77:45392) 2024-01-15T14:35:53.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7ffff700 1 -- 172.21.15.77:0/375598339 learned_addr learned my addr 172.21.15.77:0/375598339 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/375598339 >> v2:172.21.15.159:3300/0 conn(0x7eff80140f20 0x7eff800b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7effd700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 0x7eff800b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870726802 0 0) 0x7eff80161b40 con 0x7eff800b2590 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff680035b0 con 0x7eff800b2590 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 775937037 0 0) 0x7eff680035b0 con 0x7eff800b2590 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 msgr2=0x7eff800b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 0x7eff800b2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 >> v2:172.21.15.159:3300/0 conn(0x7eff80140f20 msgr2=0x7eff800b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.137+0000 7eff7e7fc700 1 --2- 172.21.15.77:0/375598339 >> v2:172.21.15.159:3300/0 conn(0x7eff80140f20 0x7eff800b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff80161d90 con 0x7eff800b2590 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7eff780019f0 con 0x7eff800b2590 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 >> v1:172.21.15.77:6789/0 conn(0x7eff800b2590 legacy=0x7eff80071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff68003a00 0x7eff68003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68004850 0x7eff68004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 0x7eff68005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff800b2a20 con 0x7eff68003a00 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff800b3e10 con 0x7eff68004850 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6800ba10 con 0x7eff800b3900 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff68003a00 0x7eff68003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7effd700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68004850 0x7eff68004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7ffff700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 0x7eff68005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 -- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 msgr2=0x7eff68005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff800b3900 0x7eff68005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 -- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68004850 msgr2=0x7eff68004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68004850 0x7eff68004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 -- 172.21.15.77:0/375598339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680035b0 con 0x7eff68003a00 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff68003a00 0x7eff68003e20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7eff74009b60 tx=0x7eff68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ad57a575f7c08d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7400e6c0 con 0x7eff68003a00 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7400e820 con 0x7eff68003a00 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.138+0000 7eff7e7fc700 1 -- 172.21.15.77:0/375598339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7400eb90 con 0x7eff68003a00 2024-01-15T14:35:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff85a68700 1 -- 172.21.15.77:0/375598339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff68003a00 msgr2=0x7eff68003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff85a68700 1 --2- 172.21.15.77:0/375598339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff68003a00 0x7eff68003e20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7eff74009b60 tx=0x7eff68005880 comp rx=0 tx=0).stop 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff7ffff700 1 -- 172.21.15.77:0/375598339 reap_dead start 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff85a68700 1 -- 172.21.15.77:0/375598339 shutdown_connections 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff85a68700 1 -- 172.21.15.77:0/375598339 >> 172.21.15.77:0/375598339 conn(0x7eff800743b0 msgr2=0x7eff8006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff85a68700 1 -- 172.21.15.77:0/375598339 shutdown_connections 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.139+0000 7eff85a68700 1 -- 172.21.15.77:0/375598339 wait complete. 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.140+0000 7eff85a68700 1 Processor -- start 2024-01-15T14:35:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.140+0000 7eff85a68700 1 -- start start 2024-01-15T14:35:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff85a68700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68003a00 0x7eff800cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff85a68700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 0x7eff8014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff85a68700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff801e7980 0x7eff800af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff85a68700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff801429e0 con 0x7eff8015dbb0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff85a68700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff800cf7a0 con 0x7eff68003a00 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff85a68700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff80141e10 con 0x7eff801e7980 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 0x7eff8014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 0x7eff8014df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55150/0 (socket says 172.21.15.77:55150) 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 -- 172.21.15.77:0/3054542910 learned_addr learned my addr 172.21.15.77:0/3054542910 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 -- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff801e7980 msgr2=0x7eff800af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7ffff700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff801e7980 0x7eff800af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68003a00 0x7eff800cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff801e7980 0x7eff800af110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 -- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68003a00 msgr2=0x7eff800cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68003a00 0x7eff800cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 -- 172.21.15.77:0/3054542910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff801420b0 con 0x7eff8015dbb0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.141+0000 7eff7effd700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 0x7eff8014df80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff700119f0 tx=0x7eff70011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.142+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70004300 con 0x7eff8015dbb0 2024-01-15T14:35:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.142+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff70004460 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.142+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff80142390 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.142+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7000f950 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.142+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8014e4c0 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.143+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff80142b90 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.143+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff7000fab0 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.143+0000 7eff67fff700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7eff6c06b350 0x7eff6c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.143+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff70089e40 con 0x7eff8015dbb0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.143+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7eff6c06b350 0x7eff6c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.144+0000 7eff7f7fe700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7eff6c06b350 0x7eff6c06d810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7eff7400a010 tx=0x7eff74008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.148+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff80142b90 con 0x7eff8015dbb0 2024-01-15T14:35:53.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.416+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7eff80067f60 con 0x7eff8015dbb0 2024-01-15T14:35:53.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.418+0000 7eff67fff700 1 -- 172.21.15.77:0/3054542910 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7eff700230a0 con 0x7eff8015dbb0 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.420+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7eff6c06b350 msgr2=0x7eff6c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.420+0000 7eff85a68700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7eff6c06b350 0x7eff6c06d810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7eff7400a010 tx=0x7eff74008030 comp rx=0 tx=0).stop 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.420+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 msgr2=0x7eff8014df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.420+0000 7eff85a68700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 0x7eff8014df80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff700119f0 tx=0x7eff70011db0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.420+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 shutdown_connections 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7eff6c06b350 0x7eff6c06d810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7eff801e7980 0x7eff800af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7eff68003a00 0x7eff800cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 --2- 172.21.15.77:0/3054542910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff8015dbb0 0x7eff8014df80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 >> 172.21.15.77:0/3054542910 conn(0x7eff800743b0 msgr2=0x7eff8015f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:53.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 shutdown_connections 2024-01-15T14:35:53.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.421+0000 7eff85a68700 1 -- 172.21.15.77:0/3054542910 wait complete. 2024-01-15T14:35:53.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-15T14:35:53.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.553+0000 7fd2a32dd700 1 Processor -- start 2024-01-15T14:35:53.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.554+0000 7fd2a32dd700 1 -- start start 2024-01-15T14:35:53.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a32dd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 0x7fd29c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a32dd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29c0b0290 con 0x7fd29c0b59f0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a32dd700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29c0b4820 con 0x7fd29c0b5dd0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a32dd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0b4b00 con 0x7fd29c0b4350 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a1079700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd29c0b59f0 0x7fd29c06db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45406/0 (socket says 172.21.15.77:45406) 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a1079700 1 -- 172.21.15.77:0/3920258324 learned_addr learned my addr 172.21.15.77:0/3920258324 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451888723 0 0) 0x7fd29c0b0290 con 0x7fd29c0b59f0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a187a700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 0x7fd29c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2880035b0 con 0x7fd29c0b59f0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095228112 0 0) 0x7fd2880035b0 con 0x7fd29c0b59f0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 msgr2=0x7fd29c0af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 0x7fd29c0af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 >> v1:172.21.15.159:6789/0 conn(0x7fd29c0b5dd0 legacy=0x7fd29c0b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0ae9b0 con 0x7fd29c0b59f0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fd298001670 con 0x7fd29c0b59f0 2024-01-15T14:35:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 >> v1:172.21.15.77:6789/0 conn(0x7fd29c0b59f0 legacy=0x7fd29c06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd288003a30 0x7fd288003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd288004850 0x7fd288004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 0x7fd288005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0b4b00 con 0x7fd288003a30 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28800bac0 con 0x7fd288004850 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28800bc80 con 0x7fd29c0b4350 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a1079700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd288003a30 0x7fd288003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a0878700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd288004850 0x7fd288004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.555+0000 7fd2a187a700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 0x7fd288005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a1079700 1 -- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 msgr2=0x7fd288005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a1079700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd29c0b4350 0x7fd288005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a1079700 1 -- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd288004850 msgr2=0x7fd288004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a1079700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd288004850 0x7fd288004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a1079700 1 -- 172.21.15.77:0/3920258324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880035b0 con 0x7fd288003a30 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a1079700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd288003a30 0x7fd288003e50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd29800c210 tx=0x7fd288005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38d1f44bdce7060 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298007c10 con 0x7fd288003a30 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29800cdb0 con 0x7fd288003a30 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd28ffff700 1 -- 172.21.15.77:0/3920258324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298005ba0 con 0x7fd288003a30 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a32dd700 1 -- 172.21.15.77:0/3920258324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd288003a30 msgr2=0x7fd288003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/3920258324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd288003a30 0x7fd288003e50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd29800c210 tx=0x7fd288005880 comp rx=0 tx=0).stop 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.556+0000 7fd2a187a700 1 -- 172.21.15.77:0/3920258324 reap_dead start 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.557+0000 7fd2a32dd700 1 -- 172.21.15.77:0/3920258324 shutdown_connections 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.557+0000 7fd2a32dd700 1 -- 172.21.15.77:0/3920258324 >> 172.21.15.77:0/3920258324 conn(0x7fd29c072990 msgr2=0x7fd29c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.557+0000 7fd2a32dd700 1 -- 172.21.15.77:0/3920258324 shutdown_connections 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.557+0000 7fd2a32dd700 1 -- 172.21.15.77:0/3920258324 wait complete. 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.558+0000 7fd2a32dd700 1 Processor -- start 2024-01-15T14:35:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.558+0000 7fd2a32dd700 1 -- start start 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.558+0000 7fd2a32dd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd288003a30 0x7fd29c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a32dd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd29c14f5a0 0x7fd29c14fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a32dd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 0x7fd29c15b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a32dd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c1518c0 con 0x7fd29c15abc0 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a32dd700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c14ff60 con 0x7fd29c14f5a0 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a32dd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c15b580 con 0x7fd288003a30 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 0x7fd29c15b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 0x7fd29c15b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55170/0 (socket says 172.21.15.77:55170) 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 -- 172.21.15.77:0/1416697392 learned_addr learned my addr 172.21.15.77:0/1416697392 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a0878700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd29c14f5a0 0x7fd29c14fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a1079700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd288003a30 0x7fd29c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 -- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd288003a30 msgr2=0x7fd29c13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd288003a30 0x7fd29c13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 -- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd29c14f5a0 msgr2=0x7fd29c14fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd29c14f5a0 0x7fd29c14fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 -- 172.21.15.77:0/1416697392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c15b7d0 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a187a700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 0x7fd29c15b040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd29c070690 tx=0x7fd294011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400e040 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400f040 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29c15bab0 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.559+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294015430 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.560+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c15c070 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.560+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c067f60 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.561+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd294015590 con 0x7fd29c15abc0 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.561+0000 7fd28dffb700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd28006b350 0x7fd28006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.562+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd294089e80 con 0x7fd29c15abc0 2024-01-15T14:35:53.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.562+0000 7fd2a1079700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd28006b350 0x7fd28006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.562+0000 7fd2a1079700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd28006b350 0x7fd28006d810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd298015040 tx=0x7fd298007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.566+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd294054f10 con 0x7fd29c15abc0 2024-01-15T14:35:53.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.842+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fd29c06c2a0 con 0x7fd29c15abc0 2024-01-15T14:35:53.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.844+0000 7fd28dffb700 1 -- 172.21.15.77:0/1416697392 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd294058530 con 0x7fd29c15abc0 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd28006b350 msgr2=0x7fd28006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd28006b350 0x7fd28006d810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd298015040 tx=0x7fd298007000 comp rx=0 tx=0).stop 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 msgr2=0x7fd29c15b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 0x7fd29c15b040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd29c070690 tx=0x7fd294011f60 comp rx=0 tx=0).stop 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 shutdown_connections 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd28006b350 0x7fd28006d810 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd288003a30 0x7fd29c13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd29c15abc0 0x7fd29c15b040 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 --2- 172.21.15.77:0/1416697392 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd29c14f5a0 0x7fd29c14fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 >> 172.21.15.77:0/1416697392 conn(0x7fd29c072990 msgr2=0x7fd29c0b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 shutdown_connections 2024-01-15T14:35:53.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.846+0000 7fd2a32dd700 1 -- 172.21.15.77:0/1416697392 wait complete. 2024-01-15T14:35:53.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-15T14:35:53.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.980+0000 7fc523cef700 1 Processor -- start 2024-01-15T14:35:53.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.980+0000 7fc523cef700 1 -- start start 2024-01-15T14:35:53.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc523cef700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc51c15e2a0 0x7fc51c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc523cef700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc51c0ce090 0x7fc51c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc523cef700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c073f90 0x7fc51c074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc523cef700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51c074960 con 0x7fc51c0ce090 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc523cef700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51c06e7f0 con 0x7fc51c15e2a0 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc523cef700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c06dec0 con 0x7fc51c073f90 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc51c0ce090 0x7fc51c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc51c0ce090 0x7fc51c06ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55180/0 (socket says 172.21.15.77:55180) 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 -- 172.21.15.77:0/3773316152 learned_addr learned my addr 172.21.15.77:0/3773316152 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc521a8b700 1 --2- 172.21.15.77:0/3773316152 >> v2:172.21.15.159:3300/0 conn(0x7fc51c15e2a0 0x7fc51c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 -- 172.21.15.77:0/3773316152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c073f90 msgr2=0x7fc51c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52228c700 1 --2- 172.21.15.77:0/3773316152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c073f90 0x7fc51c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 --2- 172.21.15.77:0/3773316152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c073f90 0x7fc51c074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 -- 172.21.15.77:0/3773316152 >> v2:172.21.15.159:3300/0 conn(0x7fc51c15e2a0 msgr2=0x7fc51c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 --2- 172.21.15.77:0/3773316152 >> v2:172.21.15.159:3300/0 conn(0x7fc51c15e2a0 0x7fc51c06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 -- 172.21.15.77:0/3773316152 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c06e1a0 con 0x7fc51c0ce090 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc521a8b700 1 --2- 172.21.15.77:0/3773316152 >> v2:172.21.15.159:3300/0 conn(0x7fc51c15e2a0 0x7fc51c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc52128a700 1 --2- 172.21.15.77:0/3773316152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c06ec80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc51c06d0c0 tx=0x7fc50c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369f74dc62e55891 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc520a89700 1 -- 172.21.15.77:0/3773316152 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c00b870 con 0x7fc51c0ce090 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc520a89700 1 -- 172.21.15.77:0/3773316152 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c00b9d0 con 0x7fc51c0ce090 2024-01-15T14:35:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.981+0000 7fc520a89700 1 -- 172.21.15.77:0/3773316152 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c00bd60 con 0x7fc51c0ce090 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 -- 172.21.15.77:0/3773316152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 msgr2=0x7fc51c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 --2- 172.21.15.77:0/3773316152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c06ec80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc51c06d0c0 tx=0x7fc50c009bd0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 -- 172.21.15.77:0/3773316152 shutdown_connections 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 --2- 172.21.15.77:0/3773316152 >> v2:172.21.15.159:3300/0 conn(0x7fc51c15e2a0 0x7fc51c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 --2- 172.21.15.77:0/3773316152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c073f90 0x7fc51c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 --2- 172.21.15.77:0/3773316152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c06ec80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 -- 172.21.15.77:0/3773316152 >> 172.21.15.77:0/3773316152 conn(0x7fc51c0cd570 msgr2=0x7fc51c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 -- 172.21.15.77:0/3773316152 shutdown_connections 2024-01-15T14:35:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.982+0000 7fc523cef700 1 -- 172.21.15.77:0/3773316152 wait complete. 2024-01-15T14:35:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.983+0000 7fc523cef700 1 Processor -- start 2024-01-15T14:35:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.983+0000 7fc523cef700 1 -- start start 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc523cef700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc51c073f90 0x7fc51c14d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc523cef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c0c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc523cef700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c15e2a0 0x7fc51c0c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc523cef700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c0c8260 con 0x7fc51c0ce090 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc523cef700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c14db00 con 0x7fc51c073f90 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc523cef700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c1607f0 con 0x7fc51c15e2a0 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c0c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c0c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55196/0 (socket says 172.21.15.77:55196) 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 -- 172.21.15.77:0/3192117879 learned_addr learned my addr 172.21.15.77:0/3192117879 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52228c700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c15e2a0 0x7fc51c0c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc521a8b700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc51c073f90 0x7fc51c14d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 -- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c15e2a0 msgr2=0x7fc51c0c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c15e2a0 0x7fc51c0c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 -- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc51c073f90 msgr2=0x7fc51c14d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc51c073f90 0x7fc51c14d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52128a700 1 -- 172.21.15.77:0/3192117879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c009550 con 0x7fc51c0ce090 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc52228c700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c15e2a0 0x7fc51c0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.984+0000 7fc521a8b700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc51c073f90 0x7fc51c14d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.985+0000 7fc52128a700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c0c6740 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc51c073510 tx=0x7fc50c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.985+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c004310 con 0x7fc51c0ce090 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.985+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c01f070 con 0x7fc51c0ce090 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.985+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c160ad0 con 0x7fc51c0ce090 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.985+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0036a0 con 0x7fc51c0ce090 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.985+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51c161090 con 0x7fc51c0ce090 2024-01-15T14:35:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.986+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c14de00 con 0x7fc51c0ce090 2024-01-15T14:35:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.986+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc50c021030 con 0x7fc51c0ce090 2024-01-15T14:35:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.987+0000 7fc5127fc700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50006b350 0x7fc50006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.987+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc50c08b030 con 0x7fc51c0ce090 2024-01-15T14:35:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.987+0000 7fc521a8b700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50006b350 0x7fc50006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.987+0000 7fc521a8b700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50006b350 0x7fc50006d810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc518006fd0 tx=0x7fc518008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:53.991+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc50c056360 con 0x7fc51c0ce090 2024-01-15T14:35:54.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.259+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fc51c067f60 con 0x7fc51c0ce090 2024-01-15T14:35:54.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.261+0000 7fc5127fc700 1 -- 172.21.15.77:0/3192117879 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc50c059980 con 0x7fc51c0ce090 2024-01-15T14:35:54.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.263+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50006b350 msgr2=0x7fc50006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.263+0000 7fc523cef700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50006b350 0x7fc50006d810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc518006fd0 tx=0x7fc518008040 comp rx=0 tx=0).stop 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.263+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 msgr2=0x7fc51c0c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.263+0000 7fc523cef700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c0c6740 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc51c073510 tx=0x7fc50c004750 comp rx=0 tx=0).stop 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 shutdown_connections 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc50006b350 0x7fc50006d810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc51c15e2a0 0x7fc51c0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc51c073f90 0x7fc51c14d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 --2- 172.21.15.77:0/3192117879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51c0ce090 0x7fc51c0c6740 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 >> 172.21.15.77:0/3192117879 conn(0x7fc51c0cd570 msgr2=0x7fc51c0afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 shutdown_connections 2024-01-15T14:35:54.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.264+0000 7fc523cef700 1 -- 172.21.15.77:0/3192117879 wait complete. 2024-01-15T14:35:54.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-15T14:35:54.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.399+0000 7f3f147c8700 1 Processor -- start 2024-01-15T14:35:54.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.399+0000 7f3f147c8700 1 -- start start 2024-01-15T14:35:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f147c8700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3f0c15e2a0 0x7f3f0c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f147c8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f0c073f90 0x7f3f0c074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f147c8700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c06e7f0 con 0x7f3f0c0ce090 2024-01-15T14:35:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f147c8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c074960 con 0x7f3f0c073f90 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f147c8700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c06dec0 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f0c073f90 0x7f3f0c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f0c073f90 0x7f3f0c074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55202/0 (socket says 172.21.15.77:55202) 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 -- 172.21.15.77:0/2668978618 learned_addr learned my addr 172.21.15.77:0/2668978618 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 -- 172.21.15.77:0/2668978618 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c15e2a0 msgr2=0x7f3f0c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12564700 1 --2- 172.21.15.77:0/2668978618 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c15e2a0 0x7f3f0c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 --2- 172.21.15.77:0/2668978618 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c15e2a0 0x7f3f0c06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 -- 172.21.15.77:0/2668978618 >> v1:172.21.15.159:6789/0 conn(0x7f3f0c0ce090 legacy=0x7f3f0c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 -- 172.21.15.77:0/2668978618 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c06e1a0 con 0x7f3f0c073f90 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12564700 1 --2- 172.21.15.77:0/2668978618 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c15e2a0 0x7f3f0c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.400+0000 7f3f12d65700 1 --2- 172.21.15.77:0/2668978618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c073f90 0x7f3f0c074420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c06d0c0 tx=0x7f3f0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a2be0b4a4c3d99 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f11562700 1 -- 172.21.15.77:0/2668978618 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0400e070 con 0x7f3f0c073f90 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f11562700 1 -- 172.21.15.77:0/2668978618 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04003a20 con 0x7f3f0c073f90 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f11562700 1 -- 172.21.15.77:0/2668978618 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04003dd0 con 0x7f3f0c073f90 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 -- 172.21.15.77:0/2668978618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c073f90 msgr2=0x7f3f0c074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2668978618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c073f90 0x7f3f0c074420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c06d0c0 tx=0x7f3f0400b330 comp rx=0 tx=0).stop 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 -- 172.21.15.77:0/2668978618 shutdown_connections 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2668978618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c073f90 0x7f3f0c074420 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2668978618 >> v2:172.21.15.159:3300/0 conn(0x7f3f0c15e2a0 0x7f3f0c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 -- 172.21.15.77:0/2668978618 >> 172.21.15.77:0/2668978618 conn(0x7f3f0c0cd570 msgr2=0x7f3f0c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 -- 172.21.15.77:0/2668978618 shutdown_connections 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.401+0000 7f3f147c8700 1 -- 172.21.15.77:0/2668978618 wait complete. 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.402+0000 7f3f147c8700 1 Processor -- start 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.402+0000 7f3f147c8700 1 -- start start 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f147c8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c073f90 0x7f3f0c1f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f147c8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0ce090 0x7f3f0c0c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f147c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 0x7f3f0c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f147c8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c15f910 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f147c8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c15ed00 con 0x7f3f0c0ce090 2024-01-15T14:35:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f147c8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c15efe0 con 0x7f3f0c073f90 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f12d65700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 0x7f3f0c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f12d65700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 0x7f3f0c0b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55214/0 (socket says 172.21.15.77:55214) 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.403+0000 7f3f12d65700 1 -- 172.21.15.77:0/2687499938 learned_addr learned my addr 172.21.15.77:0/2687499938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f12d65700 1 -- 172.21.15.77:0/2687499938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c073f90 msgr2=0x7f3f0c1f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f12d65700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c073f90 0x7f3f0c1f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f12d65700 1 -- 172.21.15.77:0/2687499938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0ce090 msgr2=0x7f3f0c0c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f12d65700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0ce090 0x7f3f0c0c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f12d65700 1 -- 172.21.15.77:0/2687499938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04009b30 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f12d65700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 0x7f3f0c0b16a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3f04015040 tx=0x7f3f04004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04004240 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f040043a0 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c15f2c0 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04004750 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.404+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c150bd0 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.405+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c06bfb0 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.405+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3f04019070 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.406+0000 7f3f02ffd700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.406+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f0408a870 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.406+0000 7f3f12564700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.406+0000 7f3f12564700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f08007920 tx=0x7f3f08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.410+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f04055a40 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.681+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f3f0c067f60 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.683+0000 7f3f02ffd700 1 -- 172.21.15.77:0/2687499938 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3f04059060 con 0x7f3f0c15e2a0 2024-01-15T14:35:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 msgr2=0x7f3ef006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f08007920 tx=0x7f3f08008040 comp rx=0 tx=0).stop 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 msgr2=0x7f3f0c0b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 0x7f3f0c0b16a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3f04015040 tx=0x7f3f04004ad0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 shutdown_connections 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3ef006b350 0x7f3ef006d810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3f0c073f90 0x7f3f0c1f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0c15e2a0 0x7f3f0c0b16a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 --2- 172.21.15.77:0/2687499938 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3f0c0ce090 0x7f3f0c0c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 >> 172.21.15.77:0/2687499938 conn(0x7f3f0c0cd570 msgr2=0x7f3f0c0add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 shutdown_connections 2024-01-15T14:35:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.686+0000 7f3f147c8700 1 -- 172.21.15.77:0/2687499938 wait complete. 2024-01-15T14:35:54.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-15T14:35:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 Processor -- start 2024-01-15T14:35:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 -- start start 2024-01-15T14:35:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4c4c162920 0x7f4c4c15e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4c4c0cfbd0 0x7f4c4c15ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c161bf0 con 0x7f4c4c15bd30 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c15f450 con 0x7f4c4c162920 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.820+0000 7f4c518af700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c161e40 con 0x7f4c4c0cfbd0 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4c4c162920 0x7f4c4c15e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4c4c162920 0x7f4c4c15e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55216/0 (socket says 172.21.15.77:55216) 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 -- 172.21.15.77:0/2321646577 learned_addr learned my addr 172.21.15.77:0/2321646577 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 -- 172.21.15.77:0/2321646577 >> v2:172.21.15.159:3300/0 conn(0x7f4c4c0cfbd0 msgr2=0x7f4c4c15ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4b7fe700 1 --2- 172.21.15.77:0/2321646577 >> v2:172.21.15.159:3300/0 conn(0x7f4c4c0cfbd0 0x7f4c4c15ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 --2- 172.21.15.77:0/2321646577 >> v2:172.21.15.159:3300/0 conn(0x7f4c4c0cfbd0 0x7f4c4c15ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 -- 172.21.15.77:0/2321646577 >> v1:172.21.15.159:6789/0 conn(0x7f4c4c15bd30 legacy=0x7f4c4c0cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 -- 172.21.15.77:0/2321646577 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c162120 con 0x7f4c4c162920 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4b7fe700 1 --2- 172.21.15.77:0/2321646577 >> v2:172.21.15.159:3300/0 conn(0x7f4c4c0cfbd0 0x7f4c4c15ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c4a7fc700 1 --2- 172.21.15.77:0/2321646577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c15e6e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4c400098f0 tx=0x7f4c40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57517334eb80cc7c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c49ffb700 1 -- 172.21.15.77:0/2321646577 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40004230 con 0x7f4c4c162920 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c49ffb700 1 -- 172.21.15.77:0/2321646577 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c40004390 con 0x7f4c4c162920 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.821+0000 7f4c49ffb700 1 -- 172.21.15.77:0/2321646577 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c400036a0 con 0x7f4c4c162920 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 -- 172.21.15.77:0/2321646577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 msgr2=0x7f4c4c15e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 --2- 172.21.15.77:0/2321646577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c15e6e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4c400098f0 tx=0x7f4c40009c00 comp rx=0 tx=0).stop 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 -- 172.21.15.77:0/2321646577 shutdown_connections 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 --2- 172.21.15.77:0/2321646577 >> v2:172.21.15.159:3300/0 conn(0x7f4c4c0cfbd0 0x7f4c4c15ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 --2- 172.21.15.77:0/2321646577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c15e6e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 -- 172.21.15.77:0/2321646577 >> 172.21.15.77:0/2321646577 conn(0x7f4c4c072710 msgr2=0x7f4c4c072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 -- 172.21.15.77:0/2321646577 shutdown_connections 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.822+0000 7f4c518af700 1 -- 172.21.15.77:0/2321646577 wait complete. 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.823+0000 7f4c518af700 1 Processor -- start 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.823+0000 7f4c518af700 1 -- start start 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c518af700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c4c0cfbd0 0x7f4c4c0aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c518af700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c4c15bd30 0x7f4c4c0c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c518af700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c14df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c518af700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c50e0 con 0x7f4c4c162920 2024-01-15T14:35:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c518af700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c14e4b0 con 0x7f4c4c15bd30 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c518af700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0c44d0 con 0x7f4c4c0cfbd0 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c14df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c14df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55232/0 (socket says 172.21.15.77:55232) 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 -- 172.21.15.77:0/2750986254 learned_addr learned my addr 172.21.15.77:0/2750986254 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4a7fc700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c4c15bd30 0x7f4c4c0c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4affd700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c4c0cfbd0 0x7f4c4c0aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 -- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c4c0cfbd0 msgr2=0x7f4c4c0aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c4c0cfbd0 0x7f4c4c0aebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 -- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c4c15bd30 msgr2=0x7f4c4c0c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c4c15bd30 0x7f4c4c0c3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 -- 172.21.15.77:0/2750986254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40009550 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4affd700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c4c0cfbd0 0x7f4c4c0aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.824+0000 7f4c4b7fe700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c14df70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400cef0 tx=0x7f4c4400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.825+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400adb0 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.825+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44015440 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.825+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c0c47b0 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.825+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440157f0 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.825+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c0c4d70 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.825+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c0ccb20 con 0x7f4c4c162920 2024-01-15T14:35:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.826+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4c4400d430 con 0x7f4c4c162920 2024-01-15T14:35:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.826+0000 7f4c33fff700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4c3806b350 0x7f4c3806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.826+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4c44089e70 con 0x7f4c4c162920 2024-01-15T14:35:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.826+0000 7f4c4affd700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4c3806b350 0x7f4c3806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.827+0000 7f4c4affd700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4c3806b350 0x7f4c3806d810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c003fc0 tx=0x7f4c3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:54.831+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c44054fa0 con 0x7f4c4c162920 2024-01-15T14:35:55.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.100+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f4c4c0c5220 con 0x7f4c4c162920 2024-01-15T14:35:55.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.102+0000 7f4c33fff700 1 -- 172.21.15.77:0/2750986254 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4c4c0c5220 con 0x7f4c4c162920 2024-01-15T14:35:55.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4c3806b350 msgr2=0x7f4c3806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4c3806b350 0x7f4c3806d810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c003fc0 tx=0x7f4c3c009040 comp rx=0 tx=0).stop 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 msgr2=0x7f4c4c14df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c14df70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400cef0 tx=0x7f4c4400cf20 comp rx=0 tx=0).stop 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 shutdown_connections 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4c3806b350 0x7f4c3806d810 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4c4c0cfbd0 0x7f4c4c0aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c4c162920 0x7f4c4c14df70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 --2- 172.21.15.77:0/2750986254 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4c4c15bd30 0x7f4c4c0c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 >> 172.21.15.77:0/2750986254 conn(0x7f4c4c072710 msgr2=0x7f4c4c0cc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:55.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 shutdown_connections 2024-01-15T14:35:55.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.104+0000 7f4c518af700 1 -- 172.21.15.77:0/2750986254 wait complete. 2024-01-15T14:35:55.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-15T14:35:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.238+0000 7fc973959700 1 Processor -- start 2024-01-15T14:35:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.238+0000 7fc973959700 1 -- start start 2024-01-15T14:35:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc973959700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc96c15e080 0x7fc96c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc973959700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fc96c0ce4d0 0x7fc96c0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc973959700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96c0d0150 con 0x7fc96c0ce910 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc973959700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c15e460 con 0x7fc96c15e080 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc973959700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c0cee50 con 0x7fc96c0ce4d0 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc9716f5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc96c15e080 0x7fc96c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc971ef6700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc96c0ce910 0x7fc96c0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45408/0 (socket says 172.21.15.77:45408) 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc9716f5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc96c15e080 0x7fc96c0cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55242/0 (socket says 172.21.15.77:55242) 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc971ef6700 1 -- 172.21.15.77:0/3296977663 learned_addr learned my addr 172.21.15.77:0/3296977663 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc970ef4700 1 --2- 172.21.15.77:0/3296977663 >> v2:172.21.15.159:3300/0 conn(0x7fc96c0ce4d0 0x7fc96c0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc9716f5700 1 -- 172.21.15.77:0/3296977663 >> v2:172.21.15.159:3300/0 conn(0x7fc96c0ce4d0 msgr2=0x7fc96c0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc9716f5700 1 --2- 172.21.15.77:0/3296977663 >> v2:172.21.15.159:3300/0 conn(0x7fc96c0ce4d0 0x7fc96c0d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc9716f5700 1 -- 172.21.15.77:0/3296977663 >> v1:172.21.15.77:6789/0 conn(0x7fc96c0ce910 legacy=0x7fc96c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.239+0000 7fc9716f5700 1 -- 172.21.15.77:0/3296977663 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c0cf130 con 0x7fc96c15e080 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc9716f5700 1 --2- 172.21.15.77:0/3296977663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c0cc2a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0ce290 tx=0x7fc95c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8ef9d4a92e53b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc963fff700 1 -- 172.21.15.77:0/3296977663 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c011830 con 0x7fc96c15e080 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc963fff700 1 -- 172.21.15.77:0/3296977663 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c011990 con 0x7fc96c15e080 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc963fff700 1 -- 172.21.15.77:0/3296977663 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c011d20 con 0x7fc96c15e080 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 -- 172.21.15.77:0/3296977663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 msgr2=0x7fc96c0cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 --2- 172.21.15.77:0/3296977663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c0cc2a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0ce290 tx=0x7fc95c009b20 comp rx=0 tx=0).stop 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 -- 172.21.15.77:0/3296977663 shutdown_connections 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 --2- 172.21.15.77:0/3296977663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c0cc2a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 --2- 172.21.15.77:0/3296977663 >> v2:172.21.15.159:3300/0 conn(0x7fc96c0ce4d0 0x7fc96c0d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 -- 172.21.15.77:0/3296977663 >> 172.21.15.77:0/3296977663 conn(0x7fc96c073130 msgr2=0x7fc96c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 -- 172.21.15.77:0/3296977663 shutdown_connections 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.240+0000 7fc973959700 1 -- 172.21.15.77:0/3296977663 wait complete. 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.241+0000 7fc973959700 1 Processor -- start 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 -- start start 2024-01-15T14:35:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96c0ce4d0 0x7fc96c156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96c0ce910 0x7fc96c158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c157400 con 0x7fc96c15e080 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0c8d20 con 0x7fc96c0ce4d0 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc973959700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c159690 con 0x7fc96c0ce910 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc971ef6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc971ef6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55250/0 (socket says 172.21.15.77:55250) 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc971ef6700 1 -- 172.21.15.77:0/2167421179 learned_addr learned my addr 172.21.15.77:0/2167421179 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc9716f5700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96c0ce4d0 0x7fc96c156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.242+0000 7fc970ef4700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96c0ce910 0x7fc96c158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc971ef6700 1 -- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96c0ce910 msgr2=0x7fc96c158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc971ef6700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96c0ce910 0x7fc96c158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc971ef6700 1 -- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96c0ce4d0 msgr2=0x7fc96c156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc971ef6700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96c0ce4d0 0x7fc96c156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc971ef6700 1 -- 172.21.15.77:0/2167421179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0094a0 con 0x7fc96c15e080 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc9716f5700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96c0ce4d0 0x7fc96c156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc971ef6700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c159150 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc96400cbf0 tx=0x7fc9640021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96400ab40 con 0x7fc96c15e080 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96400aca0 con 0x7fc96c15e080 2024-01-15T14:35:55.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c0c8f70 con 0x7fc96c15e080 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964012600 con 0x7fc96c15e080 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.243+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c0c9530 con 0x7fc96c15e080 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.244+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c067f60 con 0x7fc96c15e080 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.244+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc96401b3b0 con 0x7fc96c15e080 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.245+0000 7fc961ffb700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc95006b350 0x7fc95006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.245+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc9640893c0 con 0x7fc96c15e080 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.245+0000 7fc9716f5700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc95006b350 0x7fc95006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.245+0000 7fc9716f5700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc95006b350 0x7fc95006d810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0090b0 tx=0x7fc95c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:55.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.249+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9640546c0 con 0x7fc96c15e080 2024-01-15T14:35:55.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.520+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fc96c0d19c0 con 0x7fc96c15e080 2024-01-15T14:35:55.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.522+0000 7fc961ffb700 1 -- 172.21.15.77:0/2167421179 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc964057ce0 con 0x7fc96c15e080 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc95006b350 msgr2=0x7fc95006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc95006b350 0x7fc95006d810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0090b0 tx=0x7fc95c008030 comp rx=0 tx=0).stop 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 msgr2=0x7fc96c159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c159150 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc96400cbf0 tx=0x7fc9640021c0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 shutdown_connections 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fc95006b350 0x7fc95006d810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c15e080 0x7fc96c159150 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fc96c0ce4d0 0x7fc96c156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 --2- 172.21.15.77:0/2167421179 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fc96c0ce910 0x7fc96c158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 >> 172.21.15.77:0/2167421179 conn(0x7fc96c073130 msgr2=0x7fc96c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 shutdown_connections 2024-01-15T14:35:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.524+0000 7fc973959700 1 -- 172.21.15.77:0/2167421179 wait complete. 2024-01-15T14:35:55.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.658+0000 7f7ba39da700 1 Processor -- start 2024-01-15T14:35:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.659+0000 7f7ba39da700 1 -- start start 2024-01-15T14:35:55.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba39da700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 0x7f7b9c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba39da700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c0cd100 con 0x7f7b9c15bed0 2024-01-15T14:35:55.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba39da700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c0d09a0 con 0x7f7b9c1603c0 2024-01-15T14:35:55.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba39da700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c15c2b0 con 0x7f7b9c0d2160 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba1776700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7b9c15bed0 0x7f7b9c0d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45420/0 (socket says 172.21.15.77:45420) 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba1776700 1 -- 172.21.15.77:0/3631938085 learned_addr learned my addr 172.21.15.77:0/3631938085 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113133244 0 0) 0x7f7b9c0cd100 con 0x7f7b9c15bed0 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b880035b0 con 0x7f7b9c15bed0 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 0x7f7b9c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1719181220 0 0) 0x7f7b880035b0 con 0x7f7b9c15bed0 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 msgr2=0x7f7b9c0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 0x7f7b9c0b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 >> v1:172.21.15.159:6789/0 conn(0x7f7b9c1603c0 legacy=0x7f7b9c0cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c160850 con 0x7f7b9c15bed0 2024-01-15T14:35:55.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f7b90001670 con 0x7f7b9c15bed0 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 >> v1:172.21.15.77:6789/0 conn(0x7f7b9c15bed0 legacy=0x7f7b9c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 0x7f7b88003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b88003fb0 0x7f7b88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88004d90 0x7f7b88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c15c2b0 con 0x7f7b88003fb0 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8800baf0 con 0x7f7b88004d90 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8800bcb0 con 0x7f7b9c0d2160 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b88003fb0 0x7f7b88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.660+0000 7f7ba1776700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 0x7f7b88003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba1f77700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88004d90 0x7f7b88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba0f75700 1 -- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 msgr2=0x7f7b88003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c0d2160 0x7f7b88003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba0f75700 1 -- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88004d90 msgr2=0x7f7b88005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88004d90 0x7f7b88005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba0f75700 1 -- 172.21.15.77:0/3631938085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b880035b0 con 0x7f7b88003fb0 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b88003fb0 0x7f7b88004850 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009b60 tx=0x7f7b88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1377ff4c8854473c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800e6c0 con 0x7f7b88003fb0 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9800e820 con 0x7f7b88003fb0 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7b8ffff700 1 -- 172.21.15.77:0/3631938085 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800eb90 con 0x7f7b88003fb0 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba39da700 1 -- 172.21.15.77:0/3631938085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b88003fb0 msgr2=0x7f7b88004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba39da700 1 --2- 172.21.15.77:0/3631938085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b88003fb0 0x7f7b88004850 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009b60 tx=0x7f7b88005880 comp rx=0 tx=0).stop 2024-01-15T14:35:55.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba1f77700 1 -- 172.21.15.77:0/3631938085 reap_dead start 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba39da700 1 -- 172.21.15.77:0/3631938085 shutdown_connections 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.661+0000 7f7ba39da700 1 -- 172.21.15.77:0/3631938085 >> 172.21.15.77:0/3631938085 conn(0x7f7b9c073130 msgr2=0x7f7b9c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.662+0000 7f7ba39da700 1 -- 172.21.15.77:0/3631938085 shutdown_connections 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.662+0000 7f7ba39da700 1 -- 172.21.15.77:0/3631938085 wait complete. 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.663+0000 7f7ba39da700 1 Processor -- start 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.663+0000 7f7ba39da700 1 -- start start 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88003fb0 0x7f7b9c158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 0x7f7b9c0b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c06c2a0 0x7f7b9c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c0b3330 con 0x7f7b9c0b2970 2024-01-15T14:35:55.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c06cc70 con 0x7f7b88003fb0 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c0b4ba0 con 0x7f7b9c06c2a0 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 0x7f7b9c0b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 0x7f7b9c0b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55266/0 (socket says 172.21.15.77:55266) 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 -- 172.21.15.77:0/294322567 learned_addr learned my addr 172.21.15.77:0/294322567 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba1776700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88003fb0 0x7f7b9c158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 -- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c06c2a0 msgr2=0x7f7b9c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c06c2a0 0x7f7b9c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 -- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88003fb0 msgr2=0x7f7b9c158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88003fb0 0x7f7b9c158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 -- 172.21.15.77:0/294322567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c0c9d70 con 0x7f7b9c0b2970 2024-01-15T14:35:55.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba1776700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88003fb0 0x7f7b9c158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba0f75700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 0x7f7b9c0b2df0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800e4b0 tx=0x7f7b980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800e6c0 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b98003ab0 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9c06ce80 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98003e60 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.664+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9c0ca2a0 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.665+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9c067f60 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.667+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b98013470 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.667+0000 7f7b8dffb700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b8006b350 0x7f7b8006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.667+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b9808b800 con 0x7f7b9c0b2970 2024-01-15T14:35:55.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.667+0000 7f7ba1776700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b8006b350 0x7f7b8006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:55.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.667+0000 7f7ba1776700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b8006b350 0x7f7b8006d810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c160040 tx=0x7f7b90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:55.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.671+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b98056c50 con 0x7f7b9c0b2970 2024-01-15T14:35:55.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.863+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b9c0b3330 con 0x7f7b9c0b2970 2024-01-15T14:35:55.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.864+0000 7f7b8dffb700 1 -- 172.21.15.77:0/294322567 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12857 (secure 0 0 0) 0x7f7b9c0b3330 con 0x7f7b9c0b2970 2024-01-15T14:35:55.864 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:55.864 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":30,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:52.587895+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:35:55.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b8006b350 msgr2=0x7f7b8006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b8006b350 0x7f7b8006d810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c160040 tx=0x7f7b90009040 comp rx=0 tx=0).stop 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 msgr2=0x7f7b9c0b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 0x7f7b9c0b2df0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800e4b0 tx=0x7f7b980047c0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 shutdown_connections 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7b8006b350 0x7f7b8006d810 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7b9c06c2a0 0x7f7b9c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7b88003fb0 0x7f7b9c158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 --2- 172.21.15.77:0/294322567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c0b2970 0x7f7b9c0b2df0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 >> 172.21.15.77:0/294322567 conn(0x7f7b9c073130 msgr2=0x7f7b9c0cdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 shutdown_connections 2024-01-15T14:35:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:55.866+0000 7f7ba39da700 1 -- 172.21.15.77:0/294322567 wait complete. 2024-01-15T14:35:55.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:35:55.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:35:56.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.003+0000 7fd4cebe8700 1 Processor -- start 2024-01-15T14:35:56.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.003+0000 7fd4cebe8700 1 -- start start 2024-01-15T14:35:56.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cebe8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c806e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cebe8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c806eeb0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cebe8700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c806f190 con 0x7fd4c806e9a0 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cebe8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c806f470 con 0x7fd4c815b240 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cd185700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd4c806c2e0 0x7fd4c81647f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45432/0 (socket says 172.21.15.77:45432) 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cd185700 1 -- 172.21.15.77:0/1448054363 learned_addr learned my addr 172.21.15.77:0/1448054363 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cc984700 1 --2- 172.21.15.77:0/1448054363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630976240 0 0) 0x7fd4c806eeb0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b80035b0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977391182 0 0) 0x7fd4c806f190 con 0x7fd4c806e9a0 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c806eeb0 con 0x7fd4c806e9a0 2024-01-15T14:35:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cc984700 1 -- 172.21.15.77:0/1448054363 >> v1:172.21.15.159:6789/0 conn(0x7fd4c806e9a0 legacy=0x7fd4c815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.004+0000 7fd4cc984700 1 -- 172.21.15.77:0/1448054363 >> v1:172.21.15.77:6789/0 conn(0x7fd4c806c2e0 legacy=0x7fd4c81647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cc984700 1 -- 172.21.15.77:0/1448054363 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c806f750 con 0x7fd4c815b240 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cc984700 1 --2- 172.21.15.77:0/1448054363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c806e460 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009870 tx=0x7fd4bc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47c2af34ddefd2e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc011830 con 0x7fd4c815b240 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc011990 con 0x7fd4c815b240 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4c77fe700 1 -- 172.21.15.77:0/1448054363 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc011d00 con 0x7fd4c815b240 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cebe8700 1 -- 172.21.15.77:0/1448054363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 msgr2=0x7fd4c806e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/1448054363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c806e460 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009870 tx=0x7fd4bc009b80 comp rx=0 tx=0).stop 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cebe8700 1 -- 172.21.15.77:0/1448054363 shutdown_connections 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/1448054363 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c806e460 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cebe8700 1 -- 172.21.15.77:0/1448054363 >> 172.21.15.77:0/1448054363 conn(0x7fd4c80d0c00 msgr2=0x7fd4c80d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.005+0000 7fd4cebe8700 1 -- 172.21.15.77:0/1448054363 shutdown_connections 2024-01-15T14:35:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.006+0000 7fd4cebe8700 1 -- 172.21.15.77:0/1448054363 wait complete. 2024-01-15T14:35:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.007+0000 7fd4cebe8700 1 Processor -- start 2024-01-15T14:35:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.007+0000 7fd4cebe8700 1 -- start start 2024-01-15T14:35:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cebe8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 0x7fd4c8146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cebe8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c806e9a0 0x7fd4c8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cebe8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cebe8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81f7cc0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cebe8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81f3aa0 con 0x7fd4c806e9a0 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cebe8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8146dd0 con 0x7fd4c815b240 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cc984700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 0x7fd4c8146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cc984700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 0x7fd4c8146890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55278/0 (socket says 172.21.15.77:55278) 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cc984700 1 -- 172.21.15.77:0/4153904005 learned_addr learned my addr 172.21.15.77:0/4153904005 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4cd185700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c80744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.008+0000 7fd4c7fff700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c806e9a0 0x7fd4c8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cc984700 1 -- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 msgr2=0x7fd4c80744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cc984700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c80744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cc984700 1 -- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c806e9a0 msgr2=0x7fd4c8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cc984700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c806e9a0 0x7fd4c8073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cc984700 1 -- 172.21.15.77:0/4153904005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc0094a0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cc984700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 0x7fd4c8146890 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd4c806db40 tx=0x7fd4bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc0044d0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc0036a0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc003a50 con 0x7fd4c806c2e0 2024-01-15T14:35:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c81f70b0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.009+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c81f7670 con 0x7fd4c806c2e0 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.010+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c8069020 con 0x7fd4c806c2e0 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.010+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd4bc003bb0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.011+0000 7fd4c57fa700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd4ac06b350 0x7fd4ac06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.011+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd4bc08acd0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.011+0000 7fd4c7fff700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd4ac06b350 0x7fd4ac06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.011+0000 7fd4c7fff700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd4ac06b350 0x7fd4ac06d810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd4c814cfe0 tx=0x7fd4b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.015+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4bc056040 con 0x7fd4c806c2e0 2024-01-15T14:35:56.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.204+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4c815b730 con 0x7fd4c806c2e0 2024-01-15T14:35:56.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.204+0000 7fd4c57fa700 1 -- 172.21.15.77:0/4153904005 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd4bc0150a0 con 0x7fd4c806c2e0 2024-01-15T14:35:56.204 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:35:56.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd4ac06b350 msgr2=0x7fd4ac06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd4ac06b350 0x7fd4ac06d810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd4c814cfe0 tx=0x7fd4b4009040 comp rx=0 tx=0).stop 2024-01-15T14:35:56.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 msgr2=0x7fd4c8146890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 0x7fd4c8146890 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd4c806db40 tx=0x7fd4bc004750 comp rx=0 tx=0).stop 2024-01-15T14:35:56.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 shutdown_connections 2024-01-15T14:35:56.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd4ac06b350 0x7fd4ac06d810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd4c815b240 0x7fd4c80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c806c2e0 0x7fd4c8146890 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 --2- 172.21.15.77:0/4153904005 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd4c806e9a0 0x7fd4c8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 >> 172.21.15.77:0/4153904005 conn(0x7fd4c80d0c00 msgr2=0x7fd4c815c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 shutdown_connections 2024-01-15T14:35:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.206+0000 7fd4cebe8700 1 -- 172.21.15.77:0/4153904005 wait complete. 2024-01-15T14:35:56.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T14:35:56.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.341+0000 7f964d13b700 1 Processor -- start 2024-01-15T14:35:56.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.341+0000 7f964d13b700 1 -- start start 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964d13b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 0x7f964815ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964d13b700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9648161b40 con 0x7f96480b2050 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964d13b700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480b3e10 con 0x7f9648140f20 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964d13b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9648161d90 con 0x7f96480b24d0 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f96480b2050 0x7f964815e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45442/0 (socket says 172.21.15.77:45442) 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964659c700 1 -- 172.21.15.77:0/1771823362 learned_addr learned my addr 172.21.15.77:0/1771823362 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:56.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964759e700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 0x7f964815ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406179952 0 0) 0x7f9648161b40 con 0x7f96480b2050 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96300035b0 con 0x7f96480b2050 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4176906291 0 0) 0x7f96300035b0 con 0x7f96480b2050 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 msgr2=0x7f964815ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 0x7f964815ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 >> v1:172.21.15.159:6789/0 conn(0x7f9648140f20 legacy=0x7f96480b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648162070 con 0x7f96480b2050 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f9638001670 con 0x7f96480b2050 2024-01-15T14:35:56.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 >> v1:172.21.15.77:6789/0 conn(0x7f96480b2050 legacy=0x7f964815e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 0x7f9630003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9630003f70 0x7f9630004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9630004d90 0x7f9630005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648161d90 con 0x7f9630004d90 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963000ba70 con 0x7f9630003f70 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963000bc30 con 0x7f96480b24d0 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f964759e700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9630004d90 0x7f9630005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.342+0000 7f9646d9d700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 0x7f9630003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964659c700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9630003f70 0x7f9630004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 -- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 msgr2=0x7f9630003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480b24d0 0x7f9630003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 -- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9630003f70 msgr2=0x7f9630004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9630003f70 0x7f9630004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 -- 172.21.15.77:0/1771823362 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300035b0 con 0x7f9630004d90 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9630004d90 0x7f9630005210 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9640009fc0 tx=0x7f9630005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dae17b9aa91506 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964000ec30 con 0x7f9630004d90 2024-01-15T14:35:56.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964000f040 con 0x7f9630004d90 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f9645d9b700 1 -- 172.21.15.77:0/1771823362 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640014480 con 0x7f9630004d90 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964d13b700 1 -- 172.21.15.77:0/1771823362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9630004d90 msgr2=0x7f9630005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964d13b700 1 --2- 172.21.15.77:0/1771823362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9630004d90 0x7f9630005210 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9640009fc0 tx=0x7f9630005880 comp rx=0 tx=0).stop 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964759e700 1 -- 172.21.15.77:0/1771823362 reap_dead start 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964d13b700 1 -- 172.21.15.77:0/1771823362 shutdown_connections 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964d13b700 1 -- 172.21.15.77:0/1771823362 >> 172.21.15.77:0/1771823362 conn(0x7f96480743b0 msgr2=0x7f964806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.343+0000 7f964d13b700 1 -- 172.21.15.77:0/1771823362 shutdown_connections 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.344+0000 7f964d13b700 1 -- 172.21.15.77:0/1771823362 wait complete. 2024-01-15T14:35:56.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.345+0000 7f964d13b700 1 Processor -- start 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.345+0000 7f964d13b700 1 -- start start 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964d13b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 0x7f96480ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964d13b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f964814f930 0x7f964815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964d13b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480cdda0 0x7f96480ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964d13b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480ce790 con 0x7f964814f550 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964d13b700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964815fdd0 con 0x7f964814f930 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964d13b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96481600b0 con 0x7f96480cdda0 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 0x7f96480ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 0x7f96480ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55288/0 (socket says 172.21.15.77:55288) 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 -- 172.21.15.77:0/3207970796 learned_addr learned my addr 172.21.15.77:0/3207970796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f964659c700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f964814f930 0x7f964815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 -- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480cdda0 msgr2=0x7f96480ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480cdda0 0x7f96480ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 -- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f964814f930 msgr2=0x7f964815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f964814f930 0x7f964815f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 -- 172.21.15.77:0/3207970796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648160390 con 0x7f964814f550 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.346+0000 7f9646d9d700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 0x7f96480ccda0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f963c00f7a0 tx=0x7f963c00fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.347+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c0052e0 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.347+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963c005440 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.347+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9648160670 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.347+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c0038c0 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.347+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9648160e70 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.348+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9648160b90 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.348+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f963c003a20 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.348+0000 7f962f7fe700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f963406b350 0x7f963406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.349+0000 7f964659c700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f963406b350 0x7f963406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.349+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f963c089630 con 0x7f964814f550 2024-01-15T14:35:56.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.349+0000 7f964659c700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f963406b350 0x7f963406d810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9638001050 tx=0x7f9638009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.353+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9648160b90 con 0x7f964814f550 2024-01-15T14:35:56.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.544+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9648160b90 con 0x7f964814f550 2024-01-15T14:35:56.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.544+0000 7f962f7fe700 1 -- 172.21.15.77:0/3207970796 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9648160b90 con 0x7f964814f550 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:35:56.545 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-01-15T14:35:56.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f963406b350 msgr2=0x7f963406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f963406b350 0x7f963406d810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9638001050 tx=0x7f9638009040 comp rx=0 tx=0).stop 2024-01-15T14:35:56.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 msgr2=0x7f96480ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 0x7f96480ccda0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f963c00f7a0 tx=0x7f963c00fb60 comp rx=0 tx=0).stop 2024-01-15T14:35:56.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 shutdown_connections 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f963406b350 0x7f963406d810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96480cdda0 0x7f96480ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f964814f550 0x7f96480ccda0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 --2- 172.21.15.77:0/3207970796 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f964814f930 0x7f964815f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 >> 172.21.15.77:0/3207970796 conn(0x7f96480743b0 msgr2=0x7f964806d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 shutdown_connections 2024-01-15T14:35:56.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.546+0000 7f964d13b700 1 -- 172.21.15.77:0/3207970796 wait complete. 2024-01-15T14:35:56.559 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T14:35:56.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T14:35:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.684+0000 7f3acde3f700 1 Processor -- start 2024-01-15T14:35:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.684+0000 7f3acde3f700 1 -- start start 2024-01-15T14:35:56.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3acde3f700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f3ac80cf100 0x7f3ac815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3acde3f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac806dea0 con 0x7f3ac8155780 2024-01-15T14:35:56.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3acde3f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac806e140 con 0x7f3ac806eee0 2024-01-15T14:35:56.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3acde3f700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac806dbd0 con 0x7f3ac80cf100 2024-01-15T14:35:56.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3ac77fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3ac8155780 0x7f3ac815c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45450/0 (socket says 172.21.15.77:45450) 2024-01-15T14:35:56.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3ac77fe700 1 -- 172.21.15.77:0/3882212927 learned_addr learned my addr 172.21.15.77:0/3882212927 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3ac7fff700 1 --2- 172.21.15.77:0/3882212927 >> v2:172.21.15.159:3300/0 conn(0x7f3ac80cf100 0x7f3ac815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3ac6ffd700 1 -- 172.21.15.77:0/3882212927 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109693609 0 0) 0x7f3ac806dea0 con 0x7f3ac8155780 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.685+0000 7f3ac6ffd700 1 -- 172.21.15.77:0/3882212927 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab40035b0 con 0x7f3ac8155780 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac7fff700 1 -- 172.21.15.77:0/3882212927 >> v1:172.21.15.159:6789/0 conn(0x7f3ac806eee0 legacy=0x7f3ac80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac7fff700 1 -- 172.21.15.77:0/3882212927 >> v1:172.21.15.77:6789/0 conn(0x7f3ac8155780 legacy=0x7f3ac815c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac6ffd700 1 -- 172.21.15.77:0/3882212927 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2035497817 0 0) 0x7f3ab40035b0 con 0x7f3ac8155780 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac7fff700 1 -- 172.21.15.77:0/3882212927 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac806e400 con 0x7f3ac80cf100 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac7fff700 1 --2- 172.21.15.77:0/3882212927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac80cf100 0x7f3ac815c9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009d90 tx=0x7f3ac000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fccc9b6bc970f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac6ffd700 1 -- 172.21.15.77:0/3882212927 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000e070 con 0x7f3ac80cf100 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac6ffd700 1 -- 172.21.15.77:0/3882212927 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac0003a20 con 0x7f3ac80cf100 2024-01-15T14:35:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.686+0000 7f3ac6ffd700 1 -- 172.21.15.77:0/3882212927 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8001670 con 0x7f3ac80cf100 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 -- 172.21.15.77:0/3882212927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac80cf100 msgr2=0x7f3ac815c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 --2- 172.21.15.77:0/3882212927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac80cf100 0x7f3ac815c9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009d90 tx=0x7f3ac000b330 comp rx=0 tx=0).stop 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 -- 172.21.15.77:0/3882212927 shutdown_connections 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 --2- 172.21.15.77:0/3882212927 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac80cf100 0x7f3ac815c9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 -- 172.21.15.77:0/3882212927 >> 172.21.15.77:0/3882212927 conn(0x7f3ac80d0af0 msgr2=0x7f3ac80d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 -- 172.21.15.77:0/3882212927 shutdown_connections 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.687+0000 7f3acde3f700 1 -- 172.21.15.77:0/3882212927 wait complete. 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.688+0000 7f3acde3f700 1 Processor -- start 2024-01-15T14:35:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.688+0000 7f3acde3f700 1 -- start start 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3acde3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 0x7f3ac8158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3acde3f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ac80cf100 0x7f3ac81ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3acde3f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac8155780 0x7f3ac8142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3acde3f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8158990 con 0x7f3ac806eee0 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3acde3f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81ef7b0 con 0x7f3ac8155780 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3acde3f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8155f00 con 0x7f3ac80cf100 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 0x7f3ac8158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 0x7f3ac8158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55304/0 (socket says 172.21.15.77:55304) 2024-01-15T14:35:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 -- 172.21.15.77:0/957504669 learned_addr learned my addr 172.21.15.77:0/957504669 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac7fff700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac8155780 0x7f3ac8142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3abffff700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ac80cf100 0x7f3ac81ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 -- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ac80cf100 msgr2=0x7f3ac81ef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ac80cf100 0x7f3ac81ef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 -- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac8155780 msgr2=0x7f3ac8142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac8155780 0x7f3ac8142760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 -- 172.21.15.77:0/957504669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0009a70 con 0x7f3ac806eee0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.689+0000 7f3ac77fe700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 0x7f3ac8158450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800c520 tx=0x7f3ab800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.691+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8004c80 con 0x7f3ac806eee0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.691+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab8004e00 con 0x7f3ac806eee0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.691+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac81561e0 con 0x7f3ac806eee0 2024-01-15T14:35:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.691+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8007310 con 0x7f3ac806eee0 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.691+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac8157c50 con 0x7f3ac806eee0 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.692+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac81577d0 con 0x7f3ac806eee0 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.693+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3ab8007470 con 0x7f3ac806eee0 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.693+0000 7f3ac4ff9700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3aa806b350 0x7f3aa806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.694+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3ab8089920 con 0x7f3ac806eee0 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.694+0000 7f3abffff700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3aa806b350 0x7f3aa806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:56.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.694+0000 7f3abffff700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3aa806b350 0x7f3aa806d810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8145d70 tx=0x7f3ab0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:56.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.697+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ab8054c50 con 0x7f3ac806eee0 2024-01-15T14:35:56.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:56.887+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3ac81577d0 con 0x7f3ac806eee0 2024-01-15T14:35:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.695+0000 7f3ac4ff9700 1 -- 172.21.15.77:0/957504669 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3ac81577d0 con 0x7f3ac806eee0 2024-01-15T14:35:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3aa806b350 msgr2=0x7f3aa806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3aa806b350 0x7f3aa806d810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8145d70 tx=0x7f3ab0009040 comp rx=0 tx=0).stop 2024-01-15T14:35:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 msgr2=0x7f3ac8158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 0x7f3ac8158450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800c520 tx=0x7f3ab800c830 comp rx=0 tx=0).stop 2024-01-15T14:35:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 shutdown_connections 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f3aa806b350 0x7f3aa806d810 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f3ac8155780 0x7f3ac8142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac806eee0 0x7f3ac8158450 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 --2- 172.21.15.77:0/957504669 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f3ac80cf100 0x7f3ac81ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.698+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 >> 172.21.15.77:0/957504669 conn(0x7f3ac80d0af0 msgr2=0x7f3ac80ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.699+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 shutdown_connections 2024-01-15T14:35:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.699+0000 7f3acde3f700 1 -- 172.21.15.77:0/957504669 wait complete. 2024-01-15T14:35:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-01-15T14:35:57.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T14:35:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.846+0000 7fde47760700 1 Processor -- start 2024-01-15T14:35:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.846+0000 7fde47760700 1 -- start start 2024-01-15T14:35:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde47760700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde400d05b0 0x7fde4006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde47760700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde400add40 0x7fde4015f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde47760700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde400b3d60 con 0x7fde4015ae00 2024-01-15T14:35:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde47760700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4015f8b0 con 0x7fde400d05b0 2024-01-15T14:35:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde47760700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4015b1e0 con 0x7fde400add40 2024-01-15T14:35:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde44cfb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde400d05b0 0x7fde4006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde454fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fde4015ae00 0x7fde400cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45452/0 (socket says 172.21.15.77:45452) 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde454fc700 1 -- 172.21.15.77:0/513147903 learned_addr learned my addr 172.21.15.77:0/513147903 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde45cfd700 1 --2- 172.21.15.77:0/513147903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde400add40 0x7fde4015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde44cfb700 1 -- 172.21.15.77:0/513147903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde400add40 msgr2=0x7fde4015f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.847+0000 7fde44cfb700 1 --2- 172.21.15.77:0/513147903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde400add40 0x7fde4015f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde44cfb700 1 -- 172.21.15.77:0/513147903 >> v1:172.21.15.77:6789/0 conn(0x7fde4015ae00 legacy=0x7fde400cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde37fff700 1 -- 172.21.15.77:0/513147903 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121984392 0 0) 0x7fde400b3d60 con 0x7fde4015ae00 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde44cfb700 1 -- 172.21.15.77:0/513147903 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde400b4040 con 0x7fde400d05b0 2024-01-15T14:35:57.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde44cfb700 1 --2- 172.21.15.77:0/513147903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4006dc60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00bb10 tx=0x7fde3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358f45674beec0a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde37fff700 1 -- 172.21.15.77:0/513147903 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00dcf0 con 0x7fde400d05b0 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde37fff700 1 -- 172.21.15.77:0/513147903 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c00de50 con 0x7fde400d05b0 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde37fff700 1 -- 172.21.15.77:0/513147903 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c010690 con 0x7fde400d05b0 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 -- 172.21.15.77:0/513147903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 msgr2=0x7fde4006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 --2- 172.21.15.77:0/513147903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4006dc60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00bb10 tx=0x7fde3c00be20 comp rx=0 tx=0).stop 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 -- 172.21.15.77:0/513147903 shutdown_connections 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 --2- 172.21.15.77:0/513147903 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde400add40 0x7fde4015f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 --2- 172.21.15.77:0/513147903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4006dc60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 -- 172.21.15.77:0/513147903 >> 172.21.15.77:0/513147903 conn(0x7fde40072b20 msgr2=0x7fde4006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 -- 172.21.15.77:0/513147903 shutdown_connections 2024-01-15T14:35:57.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.848+0000 7fde47760700 1 -- 172.21.15.77:0/513147903 wait complete. 2024-01-15T14:35:57.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 Processor -- start 2024-01-15T14:35:57.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 -- start start 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde400add40 0x7fde4014bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde4015ae00 0x7fde4015cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4015e230 con 0x7fde400d05b0 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4014c530 con 0x7fde400add40 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.850+0000 7fde47760700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4014a8a0 con 0x7fde4015ae00 2024-01-15T14:35:57.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55320/0 (socket says 172.21.15.77:55320) 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 -- 172.21.15.77:0/2295252039 learned_addr learned my addr 172.21.15.77:0/2295252039 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 -- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde4015ae00 msgr2=0x7fde4015cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde454fc700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde400add40 0x7fde4014bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde4015ae00 0x7fde4015cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 -- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde400add40 msgr2=0x7fde4014bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde400add40 0x7fde4014bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 -- 172.21.15.77:0/2295252039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c00b740 con 0x7fde400d05b0 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde44cfb700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4014a360 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00ba60 tx=0x7fde3c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:57.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c019810 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c019970 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4015df50 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c019d20 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.851+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4014b7f0 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.852+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde400aee90 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.853+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fde3c00f3d0 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.853+0000 7fde35ffb700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fde2406b350 0x7fde2406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.854+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fde3c08b5f0 con 0x7fde400d05b0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.854+0000 7fde454fc700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fde2406b350 0x7fde2406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:57.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.854+0000 7fde454fc700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fde2406b350 0x7fde2406d810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fde30000f60 tx=0x7fde30009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:57.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:57.858+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde3c056630 con 0x7fde400d05b0 2024-01-15T14:35:58.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.049+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fde400b44b0 con 0x7fde400d05b0 2024-01-15T14:35:58.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.698+0000 7fde35ffb700 1 -- 172.21.15.77:0/2295252039 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7fde400b44b0 con 0x7fde400d05b0 2024-01-15T14:35:58.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fde2406b350 msgr2=0x7fde2406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:58.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fde2406b350 0x7fde2406d810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fde30000f60 tx=0x7fde30009040 comp rx=0 tx=0).stop 2024-01-15T14:35:58.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 msgr2=0x7fde4014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:58.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4014a360 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00ba60 tx=0x7fde3c0047c0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 shutdown_connections 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fde2406b350 0x7fde2406d810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fde4015ae00 0x7fde4015cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fde400add40 0x7fde4014bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 --2- 172.21.15.77:0/2295252039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde400d05b0 0x7fde4014a360 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.700+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 >> 172.21.15.77:0/2295252039 conn(0x7fde40072b20 msgr2=0x7fde400b52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.701+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 shutdown_connections 2024-01-15T14:35:58.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.701+0000 7fde47760700 1 -- 172.21.15.77:0/2295252039 wait complete. 2024-01-15T14:35:58.704 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-01-15T14:35:58.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T14:35:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 Processor -- start 2024-01-15T14:35:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 -- start start 2024-01-15T14:35:58.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7ff97015e110 0x7ff97006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 0x7ff97006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97006d980 con 0x7ff97006b530 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9700affb0 con 0x7ff97015e110 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.853+0000 7ff976635700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700b0260 con 0x7ff9700afb30 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff974bd2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff97006b530 0x7ff97006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45458/0 (socket says 172.21.15.77:45458) 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff974bd2700 1 -- 172.21.15.77:0/613201887 learned_addr learned my addr 172.21.15.77:0/613201887 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678654696 0 0) 0x7ff97006d980 con 0x7ff97006b530 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96ffff700 1 --2- 172.21.15.77:0/613201887 >> v2:172.21.15.159:3300/0 conn(0x7ff97015e110 0x7ff97006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 0x7ff97006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9580035b0 con 0x7ff97006b530 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3083378101 0 0) 0x7ff9580035b0 con 0x7ff97006b530 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 msgr2=0x7ff97006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 0x7ff97006aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 >> v2:172.21.15.159:3300/0 conn(0x7ff97015e110 msgr2=0x7ff97006f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 --2- 172.21.15.77:0/613201887 >> v2:172.21.15.159:3300/0 conn(0x7ff97015e110 0x7ff97006f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700b0540 con 0x7ff97006b530 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7ff9680019f0 con 0x7ff97006b530 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 >> v1:172.21.15.77:6789/0 conn(0x7ff97006b530 legacy=0x7ff97006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 0x7ff958003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff958004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff958004d90 0x7ff958005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700affb0 con 0x7ff958003f80 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700b0260 con 0x7ff958004d90 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95800ba10 con 0x7ff9700afb30 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff958004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff96ffff700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 0x7ff958003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.854+0000 7ff974bd2700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff958004d90 0x7ff958005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96f7fe700 1 -- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 msgr2=0x7ff958003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9700afb30 0x7ff958003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96f7fe700 1 -- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff958004d90 msgr2=0x7ff958005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff958004d90 0x7ff958005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96f7fe700 1 -- 172.21.15.77:0/613201887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580035b0 con 0x7ff958003f80 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff958004850 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff960009b60 tx=0x7ff958005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f5d82670ad8e6d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960017a50 con 0x7ff958003f80 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff960017bb0 con 0x7ff958003f80 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff96effd700 1 -- 172.21.15.77:0/613201887 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000e660 con 0x7ff958003f80 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff976635700 1 -- 172.21.15.77:0/613201887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 msgr2=0x7ff958004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff976635700 1 --2- 172.21.15.77:0/613201887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff958004850 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff960009b60 tx=0x7ff958005880 comp rx=0 tx=0).stop 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff974bd2700 1 -- 172.21.15.77:0/613201887 reap_dead start 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff976635700 1 -- 172.21.15.77:0/613201887 shutdown_connections 2024-01-15T14:35:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.855+0000 7ff976635700 1 -- 172.21.15.77:0/613201887 >> 172.21.15.77:0/613201887 conn(0x7ff9700b3250 msgr2=0x7ff9700d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.856+0000 7ff976635700 1 -- 172.21.15.77:0/613201887 shutdown_connections 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.856+0000 7ff976635700 1 -- 172.21.15.77:0/613201887 wait complete. 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.857+0000 7ff976635700 1 Processor -- start 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.857+0000 7ff976635700 1 -- start start 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff9701557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff97015baa0 0x7ff97015bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701602b0 0x7ff970160760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff970156c90 con 0x7ff958003f80 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97009b820 con 0x7ff97015baa0 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97015c460 con 0x7ff9701602b0 2024-01-15T14:35:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff9701557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff9701557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55324/0 (socket says 172.21.15.77:55324) 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 -- 172.21.15.77:0/3917726757 learned_addr learned my addr 172.21.15.77:0/3917726757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff97015baa0 0x7ff97015bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff974bd2700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701602b0 0x7ff970160760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 -- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701602b0 msgr2=0x7ff970160760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701602b0 0x7ff970160760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 -- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff97015baa0 msgr2=0x7ff97015bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff97015baa0 0x7ff97015bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 -- 172.21.15.77:0/3917726757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff970156050 con 0x7ff958003f80 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96ffff700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff9701557c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff964011aa0 tx=0x7ff964011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96400f940 con 0x7ff958003f80 2024-01-15T14:35:58.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96400faa0 con 0x7ff958003f80 2024-01-15T14:35:58.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9701562f0 con 0x7ff958003f80 2024-01-15T14:35:58.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96400fe50 con 0x7ff958003f80 2024-01-15T14:35:58.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.858+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9701568b0 con 0x7ff958003f80 2024-01-15T14:35:58.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.859+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970156f00 con 0x7ff958003f80 2024-01-15T14:35:58.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.860+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff964014440 con 0x7ff958003f80 2024-01-15T14:35:58.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.860+0000 7ff96cff9700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff95c06b350 0x7ff95c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:58.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.860+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff96408a3f0 con 0x7ff958003f80 2024-01-15T14:35:58.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.861+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff95c06b350 0x7ff95c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:58.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.861+0000 7ff96f7fe700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff95c06b350 0x7ff95c06d810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9600090b0 tx=0x7ff96001b000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:58.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:58.865+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9640552b0 con 0x7ff958003f80 2024-01-15T14:35:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.103+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff970069020 con 0x7ff958003f80 2024-01-15T14:35:59.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.708+0000 7ff96cff9700 1 -- 172.21.15.77:0/3917726757 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff9640588d0 con 0x7ff958003f80 2024-01-15T14:35:59.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff95c06b350 msgr2=0x7ff95c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff95c06b350 0x7ff95c06d810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9600090b0 tx=0x7ff96001b000 comp rx=0 tx=0).stop 2024-01-15T14:35:59.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 msgr2=0x7ff9701557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff9701557c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff964011aa0 tx=0x7ff964011db0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 shutdown_connections 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff95c06b350 0x7ff95c06d810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff9701602b0 0x7ff970160760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff958003f80 0x7ff9701557c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 --2- 172.21.15.77:0/3917726757 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff97015baa0 0x7ff97015bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 >> 172.21.15.77:0/3917726757 conn(0x7ff9700b3250 msgr2=0x7ff970073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 shutdown_connections 2024-01-15T14:35:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.713+0000 7ff976635700 1 -- 172.21.15.77:0/3917726757 wait complete. 2024-01-15T14:35:59.718 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 4 and data pool 5 2024-01-15T14:35:59.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:35:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.858+0000 7f39ca4cc700 1 Processor -- start 2024-01-15T14:35:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.858+0000 7f39ca4cc700 1 -- start start 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39ca4cc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 0x7f39c415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39ca4cc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c415eb00 con 0x7f39c41473a0 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39ca4cc700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c415ede0 con 0x7f39c406dc20 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39ca4cc700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c415f0c0 con 0x7f39c4147780 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c3fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f39c41473a0 0x7f39c406c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54152/0 (socket says 172.21.15.77:54152) 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c3fff700 1 -- 172.21.15.77:0/4205415139 learned_addr learned my addr 172.21.15.77:0/4205415139 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497176306 0 0) 0x7f39c415eb00 con 0x7f39c41473a0 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac0035b0 con 0x7f39c41473a0 2024-01-15T14:35:59.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c37fe700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 0x7f39c415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972598887 0 0) 0x7f39ac0035b0 con 0x7f39c41473a0 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 msgr2=0x7f39c415e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 0x7f39c415e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 >> v1:172.21.15.159:6789/0 conn(0x7f39c406dc20 legacy=0x7f39c4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c415f3a0 con 0x7f39c41473a0 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f39b4001670 con 0x7f39c41473a0 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 >> v1:172.21.15.77:6789/0 conn(0x7f39c41473a0 legacy=0x7f39c406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 0x7f39ac003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39ac004060 0x7f39ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39ac004d90 0x7f39ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c415f0c0 con 0x7f39ac004d90 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac00bbd0 con 0x7f39ac004060 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac00bd90 con 0x7f39c4147780 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c8a69700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39ac004d90 0x7f39ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.859+0000 7f39c3fff700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 0x7f39ac003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c37fe700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39ac004060 0x7f39ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 -- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 msgr2=0x7f39ac003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c4147780 0x7f39ac003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 -- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39ac004060 msgr2=0x7f39ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39ac004060 0x7f39ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 -- 172.21.15.77:0/4205415139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0035b0 con 0x7f39ac004d90 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39ac004d90 0x7f39ac005210 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00c190 tx=0x7f39ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b533c79de213e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc001db0 con 0x7f39ac004d90 2024-01-15T14:35:59.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc00ccd0 con 0x7f39ac004d90 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c2ffd700 1 -- 172.21.15.77:0/4205415139 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc003c40 con 0x7f39ac004d90 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39ca4cc700 1 -- 172.21.15.77:0/4205415139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39ac004d90 msgr2=0x7f39ac005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/4205415139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39ac004d90 0x7f39ac005210 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00c190 tx=0x7f39ac005880 comp rx=0 tx=0).stop 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.860+0000 7f39c8a69700 1 -- 172.21.15.77:0/4205415139 reap_dead start 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.861+0000 7f39ca4cc700 1 -- 172.21.15.77:0/4205415139 shutdown_connections 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.861+0000 7f39ca4cc700 1 -- 172.21.15.77:0/4205415139 >> 172.21.15.77:0/4205415139 conn(0x7f39c40b3130 msgr2=0x7f39c40d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.861+0000 7f39ca4cc700 1 -- 172.21.15.77:0/4205415139 shutdown_connections 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.861+0000 7f39ca4cc700 1 -- 172.21.15.77:0/4205415139 wait complete. 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.862+0000 7f39ca4cc700 1 Processor -- start 2024-01-15T14:35:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.862+0000 7f39ca4cc700 1 -- start start 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c40cfe80 0x7f39c413f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 0x7f39c41f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c40ae8c0 0x7f39c40aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c413fea0 con 0x7f39c413def0 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40af280 con 0x7f39c40cfe80 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c4140640 con 0x7f39c40ae8c0 2024-01-15T14:35:59.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 0x7f39c41f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 0x7f39c41f4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57456/0 (socket says 172.21.15.77:57456) 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 -- 172.21.15.77:0/2245288978 learned_addr learned my addr 172.21.15.77:0/2245288978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c8a69700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c40ae8c0 0x7f39c40aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c3fff700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c40cfe80 0x7f39c413f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 -- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c40ae8c0 msgr2=0x7f39c40aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c40ae8c0 0x7f39c40aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 -- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c40cfe80 msgr2=0x7f39c413f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c40cfe80 0x7f39c413f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 -- 172.21.15.77:0/2245288978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c41f44d0 con 0x7f39c413def0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c37fe700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 0x7f39c41f4960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f39b8011650 tx=0x7f39b8011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800c070 con 0x7f39c413def0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b8015870 con 0x7f39c413def0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c4140080 con 0x7f39c413def0 2024-01-15T14:35:59.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8015c20 con 0x7f39c413def0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.863+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c41f4160 con 0x7f39c413def0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.865+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f39b8013400 con 0x7f39c413def0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.865+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c41f4660 con 0x7f39c413def0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.865+0000 7f39c0ff9700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f39b006b350 0x7f39b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.865+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39b8089e40 con 0x7f39c413def0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.865+0000 7f39c3fff700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f39b006b350 0x7f39b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.866+0000 7f39c3fff700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f39b006b350 0x7f39b006d810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39b4001050 tx=0x7f39b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:35:59.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:35:59.870+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39b8054cb0 con 0x7f39c413def0 2024-01-15T14:36:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.062+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39c4074ac0 con 0x7f39c413def0 2024-01-15T14:36:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.063+0000 7f39c0ff9700 1 -- 172.21.15.77:0/2245288978 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f39b80582d0 con 0x7f39c413def0 2024-01-15T14:36:00.065 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:00.065 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:35:59.105364+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:56.905929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:58.067565+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f39b006b350 msgr2=0x7f39b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f39b006b350 0x7f39b006d810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39b4001050 tx=0x7f39b4009040 comp rx=0 tx=0).stop 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 msgr2=0x7f39c41f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 0x7f39c41f4960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f39b8011650 tx=0x7f39b8011a10 comp rx=0 tx=0).stop 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 shutdown_connections 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f39b006b350 0x7f39b006d810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f39c40ae8c0 0x7f39c40aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f39c40cfe80 0x7f39c413f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 --2- 172.21.15.77:0/2245288978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c413def0 0x7f39c41f4960 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 >> 172.21.15.77:0/2245288978 conn(0x7f39c40b3130 msgr2=0x7f39c406b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:00.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 shutdown_connections 2024-01-15T14:36:00.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.065+0000 7f39ca4cc700 1 -- 172.21.15.77:0/2245288978 wait complete. 2024-01-15T14:36:00.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:36:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.201+0000 7f7e9aaa0700 1 Processor -- start 2024-01-15T14:36:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.201+0000 7f7e9aaa0700 1 -- start start 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9aaa0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e940cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9aaa0700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f7e940b49a0 0x7f7e940d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9aaa0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e940d09a0 con 0x7f7e94152cc0 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9aaa0700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9415c4b0 con 0x7f7e940b49a0 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9aaa0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9415c790 con 0x7f7e9406c710 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9883c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7e94152cc0 0x7f7e9406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54166/0 (socket says 172.21.15.77:54166) 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9883c700 1 -- 172.21.15.77:0/2669746534 learned_addr learned my addr 172.21.15.77:0/2669746534 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:00.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948086545 0 0) 0x7f7e940d09a0 con 0x7f7e94152cc0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e840035b0 con 0x7f7e94152cc0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e9903d700 1 --2- 172.21.15.77:0/2669746534 >> v2:172.21.15.159:3300/0 conn(0x7f7e940b49a0 0x7f7e940d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e93fff700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e940cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3830427420 0 0) 0x7f7e840035b0 con 0x7f7e94152cc0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 msgr2=0x7f7e940cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.202+0000 7f7e937fe700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e940cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 >> v2:172.21.15.159:3300/0 conn(0x7f7e940b49a0 msgr2=0x7f7e940d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 --2- 172.21.15.77:0/2669746534 >> v2:172.21.15.159:3300/0 conn(0x7f7e940b49a0 0x7f7e940d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9415ca70 con 0x7f7e94152cc0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f7e88001670 con 0x7f7e94152cc0 2024-01-15T14:36:00.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e940cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 >> v1:172.21.15.77:6789/0 conn(0x7f7e94152cc0 legacy=0x7f7e9406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e84003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e84004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e84004d90 0x7f7e84005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9415c4b0 con 0x7f7e84003f40 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9415c790 con 0x7f7e84004d90 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8400bac0 con 0x7f7e9406c710 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e9883c700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e84003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e9903d700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e84004d90 0x7f7e84005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 -- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 msgr2=0x7f7e84003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e9406c710 0x7f7e84003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 -- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e84004d90 msgr2=0x7f7e84005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e84004d90 0x7f7e84005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 -- 172.21.15.77:0/2669746534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840035b0 con 0x7f7e84003f40 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e93fff700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e84004850 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009920 tx=0x7f7e84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d979fd8fca076b31 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8000c070 con 0x7f7e84003f40 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e80013610 con 0x7f7e84003f40 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.203+0000 7f7e937fe700 1 -- 172.21.15.77:0/2669746534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e800139a0 con 0x7f7e84003f40 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/2669746534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 msgr2=0x7f7e84004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/2669746534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e84004850 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009920 tx=0x7f7e84005880 comp rx=0 tx=0).stop 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9903d700 1 -- 172.21.15.77:0/2669746534 reap_dead start 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/2669746534 shutdown_connections 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/2669746534 >> 172.21.15.77:0/2669746534 conn(0x7f7e94071b90 msgr2=0x7f7e94071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/2669746534 shutdown_connections 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.204+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/2669746534 wait complete. 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.205+0000 7f7e9aaa0700 1 Processor -- start 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.205+0000 7f7e9aaa0700 1 -- start start 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9aaa0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e9414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9aaa0700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e94147680 0x7f7e941f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9aaa0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e940b1c80 0x7f7e940b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9aaa0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940b2650 con 0x7f7e84003f40 2024-01-15T14:36:00.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9aaa0700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94147b00 con 0x7f7e94147680 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9aaa0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941548c0 con 0x7f7e940b1c80 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e9414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e9414aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57472/0 (socket says 172.21.15.77:57472) 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 -- 172.21.15.77:0/1530953929 learned_addr learned my addr 172.21.15.77:0/1530953929 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e93fff700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e94147680 0x7f7e941f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9903d700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e940b1c80 0x7f7e940b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 -- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e940b1c80 msgr2=0x7f7e940b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e940b1c80 0x7f7e940b2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 -- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e94147680 msgr2=0x7f7e941f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e94147680 0x7f7e941f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 -- 172.21.15.77:0/1530953929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e941f2030 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.206+0000 7f7e9883c700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e9414aed0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e780 tx=0x7f7e8800ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.207+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88010510 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.207+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e88010690 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.207+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e941f2310 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.207+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88010a40 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.207+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e94154b60 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.208+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e940753f0 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.208+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7e88010ba0 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.209+0000 7f7e917fa700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7e7806b350 0x7f7e7806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.209+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e88089960 con 0x7f7e84003f40 2024-01-15T14:36:00.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.209+0000 7f7e93fff700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7e7806b350 0x7f7e7806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.209+0000 7f7e93fff700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7e7806b350 0x7f7e7806d810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7e80000c00 tx=0x7f7e8001e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:00.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.213+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e880549f0 con 0x7f7e84003f40 2024-01-15T14:36:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.404+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e94067f60 con 0x7f7e84003f40 2024-01-15T14:36:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.405+0000 7f7e917fa700 1 -- 172.21.15.77:0/1530953929 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f7e88023020 con 0x7f7e84003f40 2024-01-15T14:36:00.407 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:00.407 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":34,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:36:00.170893+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:56.905929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:58.067565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:36:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7e7806b350 msgr2=0x7f7e7806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7e7806b350 0x7f7e7806d810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7e80000c00 tx=0x7f7e8001e040 comp rx=0 tx=0).stop 2024-01-15T14:36:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 msgr2=0x7f7e9414aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e9414aed0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e780 tx=0x7f7e8800ea90 comp rx=0 tx=0).stop 2024-01-15T14:36:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 shutdown_connections 2024-01-15T14:36:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7e7806b350 0x7f7e7806d810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7e940b1c80 0x7f7e940b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e84003f40 0x7f7e9414aed0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 --2- 172.21.15.77:0/1530953929 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7e94147680 0x7f7e941f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 >> 172.21.15.77:0/1530953929 conn(0x7f7e94071b90 msgr2=0x7f7e940b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 shutdown_connections 2024-01-15T14:36:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.407+0000 7f7e9aaa0700 1 -- 172.21.15.77:0/1530953929 wait complete. 2024-01-15T14:36:00.421 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T14:36:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.544+0000 7f47a4cfc700 1 Processor -- start 2024-01-15T14:36:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.544+0000 7f47a4cfc700 1 -- start start 2024-01-15T14:36:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.544+0000 7f47a4cfc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 0x7f47a00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f47a4cfc700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f47a009b110 0x7f47a00d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f47a4cfc700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00d0a70 con 0x7f47a0161a70 2024-01-15T14:36:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f47a4cfc700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00d1c10 con 0x7f47a009b110 2024-01-15T14:36:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f47a4cfc700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b2ab0 con 0x7f47a0161690 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 0x7f47a00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479ed9d700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f47a009b110 0x7f47a00d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 0x7f47a00d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:57324/0 (socket says 172.21.15.77:57324) 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 -- 172.21.15.77:0/2350580973 learned_addr learned my addr 172.21.15.77:0/2350580973 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479dd9b700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f47a0161a70 0x7f47a009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:56268/0 (socket says 172.21.15.77:56268) 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 -- 172.21.15.77:0/2350580973 >> v2:172.21.15.159:3300/0 conn(0x7f47a009b110 msgr2=0x7f47a00d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 --2- 172.21.15.77:0/2350580973 >> v2:172.21.15.159:3300/0 conn(0x7f47a009b110 0x7f47a00d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 -- 172.21.15.77:0/2350580973 >> v1:172.21.15.159:6789/0 conn(0x7f47a0161a70 legacy=0x7f47a009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 -- 172.21.15.77:0/2350580973 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00b2d90 con 0x7f47a0161690 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479ed9d700 1 --2- 172.21.15.77:0/2350580973 >> v2:172.21.15.159:3300/0 conn(0x7f47a009b110 0x7f47a00d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.545+0000 7f479e59c700 1 --2- 172.21.15.77:0/2350580973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 0x7f47a00d0f20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47880098f0 tx=0x7f4788009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53c011057e72b7c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:00.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f479d59a700 1 -- 172.21.15.77:0/2350580973 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788011830 con 0x7f47a0161690 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f479d59a700 1 -- 172.21.15.77:0/2350580973 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788011990 con 0x7f47a0161690 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f479d59a700 1 -- 172.21.15.77:0/2350580973 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788011d20 con 0x7f47a0161690 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 -- 172.21.15.77:0/2350580973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 msgr2=0x7f47a00d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/2350580973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 0x7f47a00d0f20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47880098f0 tx=0x7f4788009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 -- 172.21.15.77:0/2350580973 shutdown_connections 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/2350580973 >> v2:172.21.15.159:3300/0 conn(0x7f47a009b110 0x7f47a00d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/2350580973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161690 0x7f47a00d0f20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 -- 172.21.15.77:0/2350580973 >> 172.21.15.77:0/2350580973 conn(0x7f47a00d1e50 msgr2=0x7f47a00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 -- 172.21.15.77:0/2350580973 shutdown_connections 2024-01-15T14:36:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.546+0000 7f47a4cfc700 1 -- 172.21.15.77:0/2350580973 wait complete. 2024-01-15T14:36:00.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.548+0000 7f47a4cfc700 1 Processor -- start 2024-01-15T14:36:00.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.548+0000 7f47a4cfc700 1 -- start start 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f47a4cfc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 0x7f47a01f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f47a4cfc700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47a0161690 0x7f47a0152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f47a4cfc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161a70 0x7f47a00b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f47a4cfc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0152880 con 0x7f47a009b110 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f47a4cfc700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b0f30 con 0x7f47a0161690 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f47a4cfc700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01f79b0 con 0x7f47a0161a70 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 0x7f47a01f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 0x7f47a01f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57486/0 (socket says 172.21.15.77:57486) 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 -- 172.21.15.77:0/1804052503 learned_addr learned my addr 172.21.15.77:0/1804052503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479ed9d700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161a70 0x7f47a00b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479dd9b700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47a0161690 0x7f47a0152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 -- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161a70 msgr2=0x7f47a00b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161a70 0x7f47a00b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 -- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47a0161690 msgr2=0x7f47a0152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47a0161690 0x7f47a0152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.549+0000 7f479e59c700 1 -- 172.21.15.77:0/1804052503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009a70 con 0x7f47a009b110 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.550+0000 7f479e59c700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 0x7f47a01f72b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4788009c40 tx=0x7f4788004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.550+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788004450 con 0x7f47a009b110 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.550+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478801e070 con 0x7f47a009b110 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.550+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4788009550 con 0x7f47a009b110 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.550+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880038e0 con 0x7f47a009b110 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.550+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a01f7f70 con 0x7f47a009b110 2024-01-15T14:36:00.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.551+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a0160980 con 0x7f47a009b110 2024-01-15T14:36:00.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.551+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4788003a40 con 0x7f47a009b110 2024-01-15T14:36:00.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.552+0000 7f4796ffd700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f478006b350 0x7f478006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:00.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.552+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f478808a9a0 con 0x7f47a009b110 2024-01-15T14:36:00.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.552+0000 7f479dd9b700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f478006b350 0x7f478006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:00.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.552+0000 7f479dd9b700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f478006b350 0x7f478006d810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47a00c7100 tx=0x7f4790007f10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:00.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.556+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4788055820 con 0x7f47a009b110 2024-01-15T14:36:00.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:00.793+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f47a009b8e0 con 0x7f47a009b110 2024-01-15T14:36:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.727+0000 7f4796ffd700 1 -- 172.21.15.77:0/1804052503 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f47880200b0 con 0x7f47a009b110 2024-01-15T14:36:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f478006b350 msgr2=0x7f478006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f478006b350 0x7f478006d810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47a00c7100 tx=0x7f4790007f10 comp rx=0 tx=0).stop 2024-01-15T14:36:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 msgr2=0x7f47a01f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 0x7f47a01f72b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4788009c40 tx=0x7f4788004750 comp rx=0 tx=0).stop 2024-01-15T14:36:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 shutdown_connections 2024-01-15T14:36:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f478006b350 0x7f478006d810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f47a0161a70 0x7f47a00b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47a009b110 0x7f47a01f72b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 --2- 172.21.15.77:0/1804052503 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f47a0161690 0x7f47a0152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 >> 172.21.15.77:0/1804052503 conn(0x7f47a00d1e50 msgr2=0x7f47a009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:01.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.730+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 shutdown_connections 2024-01-15T14:36:01.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.731+0000 7f47a4cfc700 1 -- 172.21.15.77:0/1804052503 wait complete. 2024-01-15T14:36:01.747 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T14:36:01.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T14:36:01.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.876+0000 7fd580001700 1 Processor -- start 2024-01-15T14:36:01.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.876+0000 7fd580001700 1 -- start start 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.877+0000 7fd580001700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5780b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.877+0000 7fd580001700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5780ae300 con 0x7fd57815c030 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.877+0000 7fd580001700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57815fb00 con 0x7fd578073f90 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.877+0000 7fd580001700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57806e350 con 0x7fd5780d0630 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.877+0000 7fd57dd9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd57815c030 0x7fd57806dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54176/0 (socket says 172.21.15.77:54176) 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.877+0000 7fd57dd9d700 1 -- 172.21.15.77:0/1281674977 learned_addr learned my addr 172.21.15.77:0/1281674977 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57d59c700 1 --2- 172.21.15.77:0/1281674977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5780b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280349809 0 0) 0x7fd5780ae300 con 0x7fd57815c030 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5640035b0 con 0x7fd57815c030 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842904502 0 0) 0x7fd57815fb00 con 0x7fd578073f90 2024-01-15T14:36:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5780ae300 con 0x7fd578073f90 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1078646429 0 0) 0x7fd5640035b0 con 0x7fd57815c030 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57d59c700 1 -- 172.21.15.77:0/1281674977 >> v1:172.21.15.159:6789/0 conn(0x7fd578073f90 legacy=0x7fd57815ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57d59c700 1 -- 172.21.15.77:0/1281674977 >> v1:172.21.15.77:6789/0 conn(0x7fd57815c030 legacy=0x7fd57806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57d59c700 1 -- 172.21.15.77:0/1281674977 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57815c410 con 0x7fd5780d0630 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57d59c700 1 --2- 172.21.15.77:0/1281674977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5780b0220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd574009870 tx=0x7fd574009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f92f57aac18faa2b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568001670 con 0x7fd5780d0630 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd574004030 con 0x7fd5780d0630 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.878+0000 7fd57cd9b700 1 -- 172.21.15.77:0/1281674977 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5740043c0 con 0x7fd5780d0630 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 -- 172.21.15.77:0/1281674977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 msgr2=0x7fd5780b0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 --2- 172.21.15.77:0/1281674977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5780b0220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd574009870 tx=0x7fd574009b80 comp rx=0 tx=0).stop 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 -- 172.21.15.77:0/1281674977 shutdown_connections 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 --2- 172.21.15.77:0/1281674977 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5780b0220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 -- 172.21.15.77:0/1281674977 >> 172.21.15.77:0/1281674977 conn(0x7fd5780d0c00 msgr2=0x7fd5780d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:01.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 -- 172.21.15.77:0/1281674977 shutdown_connections 2024-01-15T14:36:01.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.879+0000 7fd580001700 1 -- 172.21.15.77:0/1281674977 wait complete. 2024-01-15T14:36:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.880+0000 7fd580001700 1 Processor -- start 2024-01-15T14:36:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.880+0000 7fd580001700 1 -- start start 2024-01-15T14:36:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd580001700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 0x7fd5781f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd580001700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5781f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd580001700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd57815c030 0x7fd5781f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd580001700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57815f410 con 0x7fd578073f90 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd580001700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd578159620 con 0x7fd57815c030 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd580001700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd578159900 con 0x7fd5780d0630 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd57dd9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 0x7fd5781f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd57dd9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 0x7fd5781f2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57500/0 (socket says 172.21.15.77:57500) 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd57dd9d700 1 -- 172.21.15.77:0/836108983 learned_addr learned my addr 172.21.15.77:0/836108983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd57d59c700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5781f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.881+0000 7fd57e59e700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd57815c030 0x7fd5781f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd57dd9d700 1 -- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 msgr2=0x7fd5781f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd57dd9d700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5781f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd57dd9d700 1 -- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd57815c030 msgr2=0x7fd5781f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd57dd9d700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd57815c030 0x7fd5781f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd57dd9d700 1 -- 172.21.15.77:0/836108983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740094a0 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd57dd9d700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 0x7fd5781f2b20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd56800c6b0 tx=0x7fd56800ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568004c80 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568004e00 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd578159be0 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568007310 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.882+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5781f3060 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.883+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd578069020 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.883+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd568007470 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.884+0000 7fd56e7fc700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd55c06b350 0x7fd55c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.884+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5680898f0 con 0x7fd578073f90 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.884+0000 7fd57d59c700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd55c06b350 0x7fd55c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:01.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.884+0000 7fd57d59c700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd55c06b350 0x7fd55c06d810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5740090b0 tx=0x7fd574008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:01.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:01.888+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5680547f0 con 0x7fd578073f90 2024-01-15T14:36:02.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.123+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd5780d1990 con 0x7fd578073f90 2024-01-15T14:36:02.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2024-01-15T14:36:02.584+0000 7f049d5e5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T14:36:02.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.729+0000 7fd56e7fc700 1 -- 172.21.15.77:0/836108983 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd568057e10 con 0x7fd578073f90 2024-01-15T14:36:02.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd55c06b350 msgr2=0x7fd55c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:02.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd55c06b350 0x7fd55c06d810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5740090b0 tx=0x7fd574008030 comp rx=0 tx=0).stop 2024-01-15T14:36:02.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 msgr2=0x7fd5781f2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:02.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 0x7fd5781f2b20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd56800c6b0 tx=0x7fd56800ca70 comp rx=0 tx=0).stop 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 shutdown_connections 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd55c06b350 0x7fd55c06d810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd57815c030 0x7fd5781f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd578073f90 0x7fd5781f2b20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 --2- 172.21.15.77:0/836108983 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd5780d0630 0x7fd5781f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 >> 172.21.15.77:0/836108983 conn(0x7fd5780d0c00 msgr2=0x7fd5781580b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 shutdown_connections 2024-01-15T14:36:02.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.731+0000 7fd580001700 1 -- 172.21.15.77:0/836108983 wait complete. 2024-01-15T14:36:02.746 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T14:36:02.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:36:02.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 Processor -- start 2024-01-15T14:36:02.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 -- start start 2024-01-15T14:36:02.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23f015de60 0x7f23f00ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:02.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f00d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:02.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f00b99e0 con 0x7f23f00cf1e0 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f00d1250 con 0x7f23f015de60 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.870+0000 7f23f5860700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00d1530 con 0x7f23f00ced60 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23eeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23f015de60 0x7f23f00ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23ef7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f23f00cf1e0 0x7f23f00d1940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54180/0 (socket says 172.21.15.77:54180) 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23ef7fe700 1 -- 172.21.15.77:0/3434789729 learned_addr learned my addr 172.21.15.77:0/3434789729 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23eeffd700 1 -- 172.21.15.77:0/3434789729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 msgr2=0x7f23f00d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23eeffd700 1 --2- 172.21.15.77:0/3434789729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f00d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23eeffd700 1 -- 172.21.15.77:0/3434789729 >> v1:172.21.15.77:6789/0 conn(0x7f23f00cf1e0 legacy=0x7f23f00d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23edffb700 1 -- 172.21.15.77:0/3434789729 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516388865 0 0) 0x7f23f00b99e0 con 0x7f23f00cf1e0 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23eeffd700 1 -- 172.21.15.77:0/3434789729 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f00d33f0 con 0x7f23f015de60 2024-01-15T14:36:02.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23eeffd700 1 --2- 172.21.15.77:0/3434789729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00ce6a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009840 tx=0x7f23e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694735eedf309852 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23edffb700 1 -- 172.21.15.77:0/3434789729 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4011830 con 0x7f23f015de60 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23edffb700 1 -- 172.21.15.77:0/3434789729 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e4011990 con 0x7f23f015de60 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.871+0000 7f23edffb700 1 -- 172.21.15.77:0/3434789729 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4011d20 con 0x7f23f015de60 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 -- 172.21.15.77:0/3434789729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 msgr2=0x7f23f00ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 --2- 172.21.15.77:0/3434789729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00ce6a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009840 tx=0x7f23e4009b50 comp rx=0 tx=0).stop 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 -- 172.21.15.77:0/3434789729 shutdown_connections 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 --2- 172.21.15.77:0/3434789729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00ce6a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 --2- 172.21.15.77:0/3434789729 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f00d3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 -- 172.21.15.77:0/3434789729 >> 172.21.15.77:0/3434789729 conn(0x7f23f006b410 msgr2=0x7f23f006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 -- 172.21.15.77:0/3434789729 shutdown_connections 2024-01-15T14:36:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.872+0000 7f23f5860700 1 -- 172.21.15.77:0/3434789729 wait complete. 2024-01-15T14:36:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.873+0000 7f23f5860700 1 Processor -- start 2024-01-15T14:36:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.873+0000 7f23f5860700 1 -- start start 2024-01-15T14:36:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23f5860700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f01f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23f5860700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23f00cf1e0 0x7f23f00c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23f5860700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23f5860700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00cbfc0 con 0x7f23f015de60 2024-01-15T14:36:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23f5860700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f01f34a0 con 0x7f23f00cf1e0 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23f5860700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00cc260 con 0x7f23f00ced60 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57508/0 (socket says 172.21.15.77:57508) 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 -- 172.21.15.77:0/3755702971 learned_addr learned my addr 172.21.15.77:0/3755702971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23eeffd700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f01f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ee7fc700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23f00cf1e0 0x7f23f00c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 -- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 msgr2=0x7f23f01f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f01f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 -- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23f00cf1e0 msgr2=0x7f23f00c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23f00cf1e0 0x7f23f00c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 -- 172.21.15.77:0/3755702971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e40094a0 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ee7fc700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23f00cf1e0 0x7f23f00c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23ef7fe700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00cd300 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f23e800cca0 tx=0x7f23e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.874+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800ac20 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.875+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e800ad80 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.875+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8007310 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.875+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f00cc540 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.875+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f00ccb00 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.875+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f0068f10 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.876+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f23e8007470 con 0x7f23f015de60 2024-01-15T14:36:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.877+0000 7f23d7fff700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f23dc06b350 0x7f23dc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.877+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23e8089b20 con 0x7f23f015de60 2024-01-15T14:36:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.877+0000 7f23eeffd700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f23dc06b350 0x7f23dc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.877+0000 7f23eeffd700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f23dc06b350 0x7f23dc06d810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f23e40090b0 tx=0x7f23e401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:02.881+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23e8054a70 con 0x7f23f015de60 2024-01-15T14:36:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.113+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23f006bb00 con 0x7f23f015de60 2024-01-15T14:36:03.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.114+0000 7f23d7fff700 1 -- 172.21.15.77:0/3755702971 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f23e8018070 con 0x7f23f015de60 2024-01-15T14:36:03.115 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:03.115 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4882,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6833/49995002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":49995002},{"type":"v1","addr":"172.21.15.159:6833","nonce":49995002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4885,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2815696160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2815696160},{"type":"v1","addr":"172.21.15.159:6835","nonce":2815696160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5082,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3711259103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3711259103},{"type":"v1","addr":"172.21.15.77:6836","nonce":3711259103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:59.105675+0000","modified":"2024-01-15T14:36:02.124724+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1676979021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":1676979021},{"type":"v1","addr":"172.21.15.77:6837","nonce":1676979021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T14:36:03.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f23dc06b350 msgr2=0x7f23dc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f23dc06b350 0x7f23dc06d810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f23e40090b0 tx=0x7f23e401c040 comp rx=0 tx=0).stop 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 msgr2=0x7f23f00cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00cd300 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f23e800cca0 tx=0x7f23e800a060 comp rx=0 tx=0).stop 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 shutdown_connections 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f23dc06b350 0x7f23dc06d810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f23f00ced60 0x7f23f01f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f015de60 0x7f23f00cd300 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 --2- 172.21.15.77:0/3755702971 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f23f00cf1e0 0x7f23f00c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 >> 172.21.15.77:0/3755702971 conn(0x7f23f006b410 msgr2=0x7f23f00b8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 shutdown_connections 2024-01-15T14:36:03.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.116+0000 7f23f5860700 1 -- 172.21.15.77:0/3755702971 wait complete. 2024-01-15T14:36:03.119 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 17 2024-01-15T14:36:03.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:36:03.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:36:03.175 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:36:03.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 Processor -- start 2024-01-15T14:36:03.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 -- start start 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa2cc0ce4d0 0x7fa2cc0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce910 0x7fa2cc0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc0d0150 con 0x7fa2cc15e080 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2cc15e460 con 0x7fa2cc0ce4d0 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.253+0000 7fa2d1d44700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0cc9b0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2caffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa2cc0ce4d0 0x7fa2cc0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2cb7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa2cc15e080 0x7fa2cc0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54196/0 (socket says 172.21.15.77:54196) 2024-01-15T14:36:03.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2cb7fe700 1 -- 172.21.15.77:0/455886451 learned_addr learned my addr 172.21.15.77:0/455886451 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2caffd700 1 -- 172.21.15.77:0/455886451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce910 msgr2=0x7fa2cc0d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2caffd700 1 --2- 172.21.15.77:0/455886451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce910 0x7fa2cc0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2caffd700 1 -- 172.21.15.77:0/455886451 >> v1:172.21.15.77:6789/0 conn(0x7fa2cc15e080 legacy=0x7fa2cc0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2caffd700 1 -- 172.21.15.77:0/455886451 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc0cee50 con 0x7fa2cc0ce4d0 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2ca7fc700 1 -- 172.21.15.77:0/455886451 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593357319 0 0) 0x7fa2cc0d0150 con 0x7fa2cc15e080 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2caffd700 1 --2- 172.21.15.77:0/455886451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce4d0 0x7fa2cc0d1be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009870 tx=0x7fa2c0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba66a4e868117119 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2ca7fc700 1 -- 172.21.15.77:0/455886451 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0011830 con 0x7fa2cc0ce4d0 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2ca7fc700 1 -- 172.21.15.77:0/455886451 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c0011990 con 0x7fa2cc0ce4d0 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.254+0000 7fa2ca7fc700 1 -- 172.21.15.77:0/455886451 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0011d20 con 0x7fa2cc0ce4d0 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 -- 172.21.15.77:0/455886451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce4d0 msgr2=0x7fa2cc0d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/455886451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce4d0 0x7fa2cc0d1be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009870 tx=0x7fa2c0009b80 comp rx=0 tx=0).stop 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 -- 172.21.15.77:0/455886451 shutdown_connections 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/455886451 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce910 0x7fa2cc0d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/455886451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce4d0 0x7fa2cc0d1be0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 -- 172.21.15.77:0/455886451 >> 172.21.15.77:0/455886451 conn(0x7fa2cc073130 msgr2=0x7fa2cc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:03.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 -- 172.21.15.77:0/455886451 shutdown_connections 2024-01-15T14:36:03.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.255+0000 7fa2d1d44700 1 -- 172.21.15.77:0/455886451 wait complete. 2024-01-15T14:36:03.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.256+0000 7fa2d1d44700 1 Processor -- start 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.256+0000 7fa2d1d44700 1 -- start start 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2d1d44700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce4d0 0x7fa2cc1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2d1d44700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 0x7fa2cc158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2d1d44700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2cc15e080 0x7fa2cc0c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2d1d44700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0ca0b0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2d1d44700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0c8770 con 0x7fa2cc15e080 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2d1d44700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc158600 con 0x7fa2cc0ce4d0 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 0x7fa2cc158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 0x7fa2cc158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57530/0 (socket says 172.21.15.77:57530) 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 -- 172.21.15.77:0/244234857 learned_addr learned my addr 172.21.15.77:0/244234857 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 -- 172.21.15.77:0/244234857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce4d0 msgr2=0x7fa2cc1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce4d0 0x7fa2cc1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 -- 172.21.15.77:0/244234857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2cc15e080 msgr2=0x7fa2cc0c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2cc15e080 0x7fa2cc0c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 -- 172.21.15.77:0/244234857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c00094a0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2caffd700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 0x7fa2cc158bc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00090b0 tx=0x7fa2c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00036a0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c001f070 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.257+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0003a20 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.258+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2cc0ca2c0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.258+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2cc0ca880 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.258+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2cc067f60 con 0x7fa2cc0ce910 2024-01-15T14:36:03.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.259+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa2c0003b80 con 0x7fa2cc0ce910 2024-01-15T14:36:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.259+0000 7fa2b3fff700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2b806b350 0x7fa2b806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.259+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2c0021070 con 0x7fa2cc0ce910 2024-01-15T14:36:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.259+0000 7fa2cb7fe700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2b806b350 0x7fa2b806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.260+0000 7fa2cb7fe700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2b806b350 0x7fa2b806d810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc001610 tx=0x7fa2bc0079f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:03.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.264+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2c00563a0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.454+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2cc15d6a0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.455+0000 7fa2b3fff700 1 -- 172.21.15.77:0/244234857 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fa2c00599c0 con 0x7fa2cc0ce910 2024-01-15T14:36:03.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:03.456 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":34,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:36:00.170893+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T14:35:56.905929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T14:35:58.067565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:36:03.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2b806b350 msgr2=0x7fa2b806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2b806b350 0x7fa2b806d810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc001610 tx=0x7fa2bc0079f0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 msgr2=0x7fa2cc158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 0x7fa2cc158bc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00090b0 tx=0x7fa2c0004750 comp rx=0 tx=0).stop 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 shutdown_connections 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa2b806b350 0x7fa2b806d810 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2cc0ce4d0 0x7fa2cc1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2cc15e080 0x7fa2cc0c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 --2- 172.21.15.77:0/244234857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2cc0ce910 0x7fa2cc158bc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 >> 172.21.15.77:0/244234857 conn(0x7fa2cc073130 msgr2=0x7fa2cc160c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 shutdown_connections 2024-01-15T14:36:03.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.457+0000 7fa2d1d44700 1 -- 172.21.15.77:0/244234857 wait complete. 2024-01-15T14:36:03.471 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T14:36:03.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T14:36:03.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.631+0000 7fbaa9f02700 1 Processor -- start 2024-01-15T14:36:03.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.631+0000 7fbaa9f02700 1 -- start start 2024-01-15T14:36:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa9f02700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbaa415e4b0 0x7fbaa4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa9f02700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa415eb00 con 0x7fbaa41473a0 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa9f02700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa415ede0 con 0x7fbaa4147780 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa9f02700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa415e910 con 0x7fbaa415e4b0 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbaa415e4b0 0x7fbaa4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa37fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbaa41473a0 0x7fbaa406c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54206/0 (socket says 172.21.15.77:54206) 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa37fe700 1 -- 172.21.15.77:0/977561732 learned_addr learned my addr 172.21.15.77:0/977561732 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa3fff700 1 -- 172.21.15.77:0/977561732 >> v1:172.21.15.159:6789/0 conn(0x7fbaa4147780 legacy=0x7fbaa406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa3fff700 1 -- 172.21.15.77:0/977561732 >> v1:172.21.15.77:6789/0 conn(0x7fbaa41473a0 legacy=0x7fbaa406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa3fff700 1 -- 172.21.15.77:0/977561732 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa415f0c0 con 0x7fbaa415e4b0 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa27fc700 1 -- 172.21.15.77:0/977561732 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284134727 0 0) 0x7fbaa415eb00 con 0x7fbaa41473a0 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa3fff700 1 --2- 172.21.15.77:0/977561732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa415e4b0 0x7fbaa4073f90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009d90 tx=0x7fba9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2935801f4a079387 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa27fc700 1 -- 172.21.15.77:0/977561732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00e070 con 0x7fbaa415e4b0 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa27fc700 1 -- 172.21.15.77:0/977561732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9c00bbd0 con 0x7fbaa415e4b0 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.632+0000 7fbaa27fc700 1 -- 172.21.15.77:0/977561732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c012430 con 0x7fbaa415e4b0 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 -- 172.21.15.77:0/977561732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa415e4b0 msgr2=0x7fbaa4073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/977561732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa415e4b0 0x7fbaa4073f90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009d90 tx=0x7fba9c00b330 comp rx=0 tx=0).stop 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 -- 172.21.15.77:0/977561732 shutdown_connections 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/977561732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa415e4b0 0x7fbaa4073f90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 -- 172.21.15.77:0/977561732 >> 172.21.15.77:0/977561732 conn(0x7fbaa40b3130 msgr2=0x7fbaa40d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 -- 172.21.15.77:0/977561732 shutdown_connections 2024-01-15T14:36:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.633+0000 7fbaa9f02700 1 -- 172.21.15.77:0/977561732 wait complete. 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.634+0000 7fbaa9f02700 1 Processor -- start 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.634+0000 7fbaa9f02700 1 -- start start 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa9f02700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa41473a0 0x7fbaa40b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa9f02700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 0x7fbaa415d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa9f02700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa415e4b0 0x7fbaa40ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa9f02700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40b19f0 con 0x7fbaa4147780 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa9f02700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40b1020 con 0x7fbaa41473a0 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa9f02700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40b0410 con 0x7fbaa415e4b0 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 0x7fbaa415d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 0x7fbaa415d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57564/0 (socket says 172.21.15.77:57564) 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa2ffd700 1 -- 172.21.15.77:0/4117459527 learned_addr learned my addr 172.21.15.77:0/4117459527 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa37fe700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa41473a0 0x7fbaa40b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa3fff700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa415e4b0 0x7fbaa40ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa2ffd700 1 -- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa415e4b0 msgr2=0x7fbaa40ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.635+0000 7fbaa2ffd700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa415e4b0 0x7fbaa40ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fbaa2ffd700 1 -- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa41473a0 msgr2=0x7fbaa40b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fbaa2ffd700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa41473a0 0x7fbaa40b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fbaa2ffd700 1 -- 172.21.15.77:0/4117459527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c009a70 con 0x7fbaa4147780 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fbaa2ffd700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 0x7fbaa415d190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c4b0 tx=0x7fba9800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980052d0 con 0x7fbaa4147780 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98005430 con 0x7fbaa4147780 2024-01-15T14:36:03.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa40b06f0 con 0x7fbaa4147780 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980038c0 con 0x7fbaa4147780 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.636+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa40b0cb0 con 0x7fbaa4147780 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.637+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa4067f60 con 0x7fbaa4147780 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.637+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fba98003a20 con 0x7fbaa4147780 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.638+0000 7fba8bfff700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fba9006b350 0x7fba9006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.638+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba98088ee0 con 0x7fbaa4147780 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.638+0000 7fbaa37fe700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fba9006b350 0x7fba9006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.638+0000 7fbaa37fe700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fba9006b350 0x7fba9006d810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba94007b10 tx=0x7fba94007a60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.642+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba98053d50 con 0x7fbaa4147780 2024-01-15T14:36:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.830+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbaa40b4e00 con 0x7fba9006b350 2024-01-15T14:36:03.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.839+0000 7fba8bfff700 1 -- 172.21.15.77:0/4117459527 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289253 (secure 0 0 0) 0x7fbaa40b4e00 con 0x7fba9006b350 2024-01-15T14:36:03.845 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:03.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fba9006b350 msgr2=0x7fba9006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fba9006b350 0x7fba9006d810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba94007b10 tx=0x7fba94007a60 comp rx=0 tx=0).stop 2024-01-15T14:36:03.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 msgr2=0x7fbaa415d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:03.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 0x7fbaa415d190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c4b0 tx=0x7fba9800c7c0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 shutdown_connections 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fba9006b350 0x7fba9006d810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fbaa415e4b0 0x7fbaa40ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fbaa41473a0 0x7fbaa40b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 --2- 172.21.15.77:0/4117459527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa4147780 0x7fbaa415d190 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 >> 172.21.15.77:0/4117459527 conn(0x7fbaa40b3130 msgr2=0x7fbaa4071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 shutdown_connections 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:03.842+0000 7fbaa9f02700 1 -- 172.21.15.77:0/4117459527 wait complete. 2024-01-15T14:36:03.850 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-01-15T14:36:03.855 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-01-15T14:36:03.031062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220392,"kb_used_data":4208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754328,"statfs":{"total":773094113280,"available":772868431872,"internally_reserved":0,"allocated":4308992,"data_stored":2382996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331426"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178124+0000","last_change":"2024-01-15T14:35:59.191318+0000","last_active":"2024-01-15T14:36:00.178124+0000","last_peered":"2024-01-15T14:36:00.178124+0000","last_clean":"2024-01-15T14:36:00.178124+0000","last_became_active":"2024-01-15T14:35:59.191035+0000","last_became_peered":"2024-01-15T14:35:59.191035+0000","last_unstale":"2024-01-15T14:36:00.178124+0000","last_undegraded":"2024-01-15T14:36:00.178124+0000","last_fullsized":"2024-01-15T14:36:00.178124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:22.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174537+0000","last_change":"2024-01-15T14:35:59.184745+0000","last_active":"2024-01-15T14:36:00.174537+0000","last_peered":"2024-01-15T14:36:00.174537+0000","last_clean":"2024-01-15T14:36:00.174537+0000","last_became_active":"2024-01-15T14:35:59.184484+0000","last_became_peered":"2024-01-15T14:35:59.184484+0000","last_unstale":"2024-01-15T14:36:00.174537+0000","last_undegraded":"2024-01-15T14:36:00.174537+0000","last_fullsized":"2024-01-15T14:36:00.174537+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:00.462542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345194+0000","last_change":"2024-01-15T14:35:59.189023+0000","last_active":"2024-01-15T14:36:01.345194+0000","last_peered":"2024-01-15T14:36:01.345194+0000","last_clean":"2024-01-15T14:36:01.345194+0000","last_became_active":"2024-01-15T14:35:59.187427+0000","last_became_peered":"2024-01-15T14:35:59.187427+0000","last_unstale":"2024-01-15T14:36:01.345194+0000","last_undegraded":"2024-01-15T14:36:01.345194+0000","last_fullsized":"2024-01-15T14:36:01.345194+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:30:42.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446328+0000","last_change":"2024-01-15T14:35:59.194817+0000","last_active":"2024-01-15T14:36:01.446328+0000","last_peered":"2024-01-15T14:36:01.446328+0000","last_clean":"2024-01-15T14:36:01.446328+0000","last_became_active":"2024-01-15T14:35:59.194526+0000","last_became_peered":"2024-01-15T14:35:59.194526+0000","last_unstale":"2024-01-15T14:36:01.446328+0000","last_undegraded":"2024-01-15T14:36:01.446328+0000","last_fullsized":"2024-01-15T14:36:01.446328+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:30.135562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445860+0000","last_change":"2024-01-15T14:35:59.187821+0000","last_active":"2024-01-15T14:36:01.445860+0000","last_peered":"2024-01-15T14:36:01.445860+0000","last_clean":"2024-01-15T14:36:01.445860+0000","last_became_active":"2024-01-15T14:35:59.187568+0000","last_became_peered":"2024-01-15T14:35:59.187568+0000","last_unstale":"2024-01-15T14:36:01.445860+0000","last_undegraded":"2024-01-15T14:36:01.445860+0000","last_fullsized":"2024-01-15T14:36:01.445860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:33:57.617411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174422+0000","last_change":"2024-01-15T14:35:59.189045+0000","last_active":"2024-01-15T14:36:00.174422+0000","last_peered":"2024-01-15T14:36:00.174422+0000","last_clean":"2024-01-15T14:36:00.174422+0000","last_became_active":"2024-01-15T14:35:59.188788+0000","last_became_peered":"2024-01-15T14:35:59.188788+0000","last_unstale":"2024-01-15T14:36:00.174422+0000","last_undegraded":"2024-01-15T14:36:00.174422+0000","last_fullsized":"2024-01-15T14:36:00.174422+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:40.243794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178585+0000","last_change":"2024-01-15T14:35:59.192905+0000","last_active":"2024-01-15T14:36:00.178585+0000","last_peered":"2024-01-15T14:36:00.178585+0000","last_clean":"2024-01-15T14:36:00.178585+0000","last_became_active":"2024-01-15T14:35:59.192639+0000","last_became_peered":"2024-01-15T14:35:59.192639+0000","last_unstale":"2024-01-15T14:36:00.178585+0000","last_undegraded":"2024-01-15T14:36:00.178585+0000","last_fullsized":"2024-01-15T14:36:00.178585+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:40.243794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177530+0000","last_change":"2024-01-15T14:35:59.191333+0000","last_active":"2024-01-15T14:36:00.177530+0000","last_peered":"2024-01-15T14:36:00.177530+0000","last_clean":"2024-01-15T14:36:00.177530+0000","last_became_active":"2024-01-15T14:35:59.191031+0000","last_became_peered":"2024-01-15T14:35:59.191031+0000","last_unstale":"2024-01-15T14:36:00.177530+0000","last_undegraded":"2024-01-15T14:36:00.177530+0000","last_fullsized":"2024-01-15T14:36:00.177530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:30.135562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445740+0000","last_change":"2024-01-15T14:35:59.198480+0000","last_active":"2024-01-15T14:36:01.445740+0000","last_peered":"2024-01-15T14:36:01.445740+0000","last_clean":"2024-01-15T14:36:01.445740+0000","last_became_active":"2024-01-15T14:35:59.197899+0000","last_became_peered":"2024-01-15T14:35:59.197899+0000","last_unstale":"2024-01-15T14:36:01.445740+0000","last_undegraded":"2024-01-15T14:36:01.445740+0000","last_fullsized":"2024-01-15T14:36:01.445740+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446451+0000","last_change":"2024-01-15T14:35:59.186140+0000","last_active":"2024-01-15T14:36:01.446451+0000","last_peered":"2024-01-15T14:36:01.446451+0000","last_clean":"2024-01-15T14:36:01.446451+0000","last_became_active":"2024-01-15T14:35:59.185835+0000","last_became_peered":"2024-01-15T14:35:59.185835+0000","last_unstale":"2024-01-15T14:36:01.446451+0000","last_undegraded":"2024-01-15T14:36:01.446451+0000","last_fullsized":"2024-01-15T14:36:01.446451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:18:13.350946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177931+0000","last_change":"2024-01-15T14:35:59.191736+0000","last_active":"2024-01-15T14:36:00.177931+0000","last_peered":"2024-01-15T14:36:00.177931+0000","last_clean":"2024-01-15T14:36:00.177931+0000","last_became_active":"2024-01-15T14:35:59.191407+0000","last_became_peered":"2024-01-15T14:35:59.191407+0000","last_unstale":"2024-01-15T14:36:00.177931+0000","last_undegraded":"2024-01-15T14:36:00.177931+0000","last_fullsized":"2024-01-15T14:36:00.177931+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345184+0000","last_change":"2024-01-15T14:35:59.188228+0000","last_active":"2024-01-15T14:36:01.345184+0000","last_peered":"2024-01-15T14:36:01.345184+0000","last_clean":"2024-01-15T14:36:01.345184+0000","last_became_active":"2024-01-15T14:35:59.187911+0000","last_became_peered":"2024-01-15T14:35:59.187911+0000","last_unstale":"2024-01-15T14:36:01.345184+0000","last_undegraded":"2024-01-15T14:36:01.345184+0000","last_fullsized":"2024-01-15T14:36:01.345184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446112+0000","last_change":"2024-01-15T14:35:59.190235+0000","last_active":"2024-01-15T14:36:01.446112+0000","last_peered":"2024-01-15T14:36:01.446112+0000","last_clean":"2024-01-15T14:36:01.446112+0000","last_became_active":"2024-01-15T14:35:59.189962+0000","last_became_peered":"2024-01-15T14:35:59.189962+0000","last_unstale":"2024-01-15T14:36:01.446112+0000","last_undegraded":"2024-01-15T14:36:01.446112+0000","last_fullsized":"2024-01-15T14:36:01.446112+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:23:33.956747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174377+0000","last_change":"2024-01-15T14:35:59.188450+0000","last_active":"2024-01-15T14:36:00.174377+0000","last_peered":"2024-01-15T14:36:00.174377+0000","last_clean":"2024-01-15T14:36:00.174377+0000","last_became_active":"2024-01-15T14:35:59.188178+0000","last_became_peered":"2024-01-15T14:35:59.188178+0000","last_unstale":"2024-01-15T14:36:00.174377+0000","last_undegraded":"2024-01-15T14:36:00.174377+0000","last_fullsized":"2024-01-15T14:36:00.174377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:37.731830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345173+0000","last_change":"2024-01-15T14:35:59.189385+0000","last_active":"2024-01-15T14:36:01.345173+0000","last_peered":"2024-01-15T14:36:01.345173+0000","last_clean":"2024-01-15T14:36:01.345173+0000","last_became_active":"2024-01-15T14:35:59.189168+0000","last_became_peered":"2024-01-15T14:35:59.189168+0000","last_unstale":"2024-01-15T14:36:01.345173+0000","last_undegraded":"2024-01-15T14:36:01.345173+0000","last_fullsized":"2024-01-15T14:36:01.345173+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177752+0000","last_change":"2024-01-15T14:35:59.190674+0000","last_active":"2024-01-15T14:36:00.177752+0000","last_peered":"2024-01-15T14:36:00.177752+0000","last_clean":"2024-01-15T14:36:00.177752+0000","last_became_active":"2024-01-15T14:35:59.190405+0000","last_became_peered":"2024-01-15T14:35:59.190405+0000","last_unstale":"2024-01-15T14:36:00.177752+0000","last_undegraded":"2024-01-15T14:36:00.177752+0000","last_fullsized":"2024-01-15T14:36:00.177752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:18:13.350946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178430+0000","last_change":"2024-01-15T14:35:59.192979+0000","last_active":"2024-01-15T14:36:00.178430+0000","last_peered":"2024-01-15T14:36:00.178430+0000","last_clean":"2024-01-15T14:36:00.178430+0000","last_became_active":"2024-01-15T14:35:59.192705+0000","last_became_peered":"2024-01-15T14:35:59.192705+0000","last_unstale":"2024-01-15T14:36:00.178430+0000","last_undegraded":"2024-01-15T14:36:00.178430+0000","last_fullsized":"2024-01-15T14:36:00.178430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:37.731830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174645+0000","last_change":"2024-01-15T14:35:59.181560+0000","last_active":"2024-01-15T14:36:00.174645+0000","last_peered":"2024-01-15T14:36:00.174645+0000","last_clean":"2024-01-15T14:36:00.174645+0000","last_became_active":"2024-01-15T14:35:59.181308+0000","last_became_peered":"2024-01-15T14:35:59.181308+0000","last_unstale":"2024-01-15T14:36:00.174645+0000","last_undegraded":"2024-01-15T14:36:00.174645+0000","last_fullsized":"2024-01-15T14:36:00.174645+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:06:57.295266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174327+0000","last_change":"2024-01-15T14:35:59.181518+0000","last_active":"2024-01-15T14:36:00.174327+0000","last_peered":"2024-01-15T14:36:00.174327+0000","last_clean":"2024-01-15T14:36:00.174327+0000","last_became_active":"2024-01-15T14:35:59.181265+0000","last_became_peered":"2024-01-15T14:35:59.181265+0000","last_unstale":"2024-01-15T14:36:00.174327+0000","last_undegraded":"2024-01-15T14:36:00.174327+0000","last_fullsized":"2024-01-15T14:36:00.174327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:06.345458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446593+0000","last_change":"2024-01-15T14:35:59.189569+0000","last_active":"2024-01-15T14:36:01.446593+0000","last_peered":"2024-01-15T14:36:01.446593+0000","last_clean":"2024-01-15T14:36:01.446593+0000","last_became_active":"2024-01-15T14:35:59.189328+0000","last_became_peered":"2024-01-15T14:35:59.189328+0000","last_unstale":"2024-01-15T14:36:01.446593+0000","last_undegraded":"2024-01-15T14:36:01.446593+0000","last_fullsized":"2024-01-15T14:36:01.446593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:27.127703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445298+0000","last_change":"2024-01-15T14:35:59.195328+0000","last_active":"2024-01-15T14:36:01.445298+0000","last_peered":"2024-01-15T14:36:01.445298+0000","last_clean":"2024-01-15T14:36:01.445298+0000","last_became_active":"2024-01-15T14:35:59.195042+0000","last_became_peered":"2024-01-15T14:35:59.195042+0000","last_unstale":"2024-01-15T14:36:01.445298+0000","last_undegraded":"2024-01-15T14:36:01.445298+0000","last_fullsized":"2024-01-15T14:36:01.445298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:47.775492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445309+0000","last_change":"2024-01-15T14:35:59.196400+0000","last_active":"2024-01-15T14:36:01.445309+0000","last_peered":"2024-01-15T14:36:01.445309+0000","last_clean":"2024-01-15T14:36:01.445309+0000","last_became_active":"2024-01-15T14:35:59.196079+0000","last_became_peered":"2024-01-15T14:35:59.196079+0000","last_unstale":"2024-01-15T14:36:01.445309+0000","last_undegraded":"2024-01-15T14:36:01.445309+0000","last_fullsized":"2024-01-15T14:36:01.445309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:19.359918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445633+0000","last_change":"2024-01-15T14:35:59.190292+0000","last_active":"2024-01-15T14:36:01.445633+0000","last_peered":"2024-01-15T14:36:01.445633+0000","last_clean":"2024-01-15T14:36:01.445633+0000","last_became_active":"2024-01-15T14:35:59.190035+0000","last_became_peered":"2024-01-15T14:35:59.190035+0000","last_unstale":"2024-01-15T14:36:01.445633+0000","last_undegraded":"2024-01-15T14:36:01.445633+0000","last_fullsized":"2024-01-15T14:36:01.445633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:49.856872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.344966+0000","last_change":"2024-01-15T14:35:59.188960+0000","last_active":"2024-01-15T14:36:01.344966+0000","last_peered":"2024-01-15T14:36:01.344966+0000","last_clean":"2024-01-15T14:36:01.344966+0000","last_became_active":"2024-01-15T14:35:59.188673+0000","last_became_peered":"2024-01-15T14:35:59.188673+0000","last_unstale":"2024-01-15T14:36:01.344966+0000","last_undegraded":"2024-01-15T14:36:01.344966+0000","last_fullsized":"2024-01-15T14:36:01.344966+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446674+0000","last_change":"2024-01-15T14:35:59.187507+0000","last_active":"2024-01-15T14:36:01.446674+0000","last_peered":"2024-01-15T14:36:01.446674+0000","last_clean":"2024-01-15T14:36:01.446674+0000","last_became_active":"2024-01-15T14:35:59.187262+0000","last_became_peered":"2024-01-15T14:35:59.187262+0000","last_unstale":"2024-01-15T14:36:01.446674+0000","last_undegraded":"2024-01-15T14:36:01.446674+0000","last_fullsized":"2024-01-15T14:36:01.446674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:30:42.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174669+0000","last_change":"2024-01-15T14:35:59.187360+0000","last_active":"2024-01-15T14:36:00.174669+0000","last_peered":"2024-01-15T14:36:00.174669+0000","last_clean":"2024-01-15T14:36:00.174669+0000","last_became_active":"2024-01-15T14:35:59.187108+0000","last_became_peered":"2024-01-15T14:35:59.187108+0000","last_unstale":"2024-01-15T14:36:00.174669+0000","last_undegraded":"2024-01-15T14:36:00.174669+0000","last_fullsized":"2024-01-15T14:36:00.174669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:21:10.420954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446670+0000","last_change":"2024-01-15T14:35:59.196017+0000","last_active":"2024-01-15T14:36:01.446670+0000","last_peered":"2024-01-15T14:36:01.446670+0000","last_clean":"2024-01-15T14:36:01.446670+0000","last_became_active":"2024-01-15T14:35:59.195845+0000","last_became_peered":"2024-01-15T14:35:59.195845+0000","last_unstale":"2024-01-15T14:36:01.446670+0000","last_undegraded":"2024-01-15T14:36:01.446670+0000","last_fullsized":"2024-01-15T14:36:01.446670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345260+0000","last_change":"2024-01-15T14:35:59.182114+0000","last_active":"2024-01-15T14:36:01.345260+0000","last_peered":"2024-01-15T14:36:01.345260+0000","last_clean":"2024-01-15T14:36:01.345260+0000","last_became_active":"2024-01-15T14:35:59.181778+0000","last_became_peered":"2024-01-15T14:35:59.181778+0000","last_unstale":"2024-01-15T14:36:01.345260+0000","last_undegraded":"2024-01-15T14:36:01.345260+0000","last_fullsized":"2024-01-15T14:36:01.345260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:06.345458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445461+0000","last_change":"2024-01-15T14:35:59.196390+0000","last_active":"2024-01-15T14:36:01.445461+0000","last_peered":"2024-01-15T14:36:01.445461+0000","last_clean":"2024-01-15T14:36:01.445461+0000","last_became_active":"2024-01-15T14:35:59.196098+0000","last_became_peered":"2024-01-15T14:35:59.196098+0000","last_unstale":"2024-01-15T14:36:01.445461+0000","last_undegraded":"2024-01-15T14:36:01.445461+0000","last_fullsized":"2024-01-15T14:36:01.445461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:30.135562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177783+0000","last_change":"2024-01-15T14:35:59.191955+0000","last_active":"2024-01-15T14:36:00.177783+0000","last_peered":"2024-01-15T14:36:00.177783+0000","last_clean":"2024-01-15T14:36:00.177783+0000","last_became_active":"2024-01-15T14:35:59.191519+0000","last_became_peered":"2024-01-15T14:35:59.191519+0000","last_unstale":"2024-01-15T14:36:00.177783+0000","last_undegraded":"2024-01-15T14:36:00.177783+0000","last_fullsized":"2024-01-15T14:36:00.177783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178297+0000","last_change":"2024-01-15T14:35:59.190402+0000","last_active":"2024-01-15T14:36:00.178297+0000","last_peered":"2024-01-15T14:36:00.178297+0000","last_clean":"2024-01-15T14:36:00.178297+0000","last_became_active":"2024-01-15T14:35:59.190116+0000","last_became_peered":"2024-01-15T14:35:59.190116+0000","last_unstale":"2024-01-15T14:36:00.178297+0000","last_undegraded":"2024-01-15T14:36:00.178297+0000","last_fullsized":"2024-01-15T14:36:00.178297+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:04.599483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178117+0000","last_change":"2024-01-15T14:35:59.195020+0000","last_active":"2024-01-15T14:36:00.178117+0000","last_peered":"2024-01-15T14:36:00.178117+0000","last_clean":"2024-01-15T14:36:00.178117+0000","last_became_active":"2024-01-15T14:35:59.194742+0000","last_became_peered":"2024-01-15T14:35:59.194742+0000","last_unstale":"2024-01-15T14:36:00.178117+0000","last_undegraded":"2024-01-15T14:36:00.178117+0000","last_fullsized":"2024-01-15T14:36:00.178117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:06.345458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446418+0000","last_change":"2024-01-15T14:35:59.187297+0000","last_active":"2024-01-15T14:36:01.446418+0000","last_peered":"2024-01-15T14:36:01.446418+0000","last_clean":"2024-01-15T14:36:01.446418+0000","last_became_active":"2024-01-15T14:35:59.186991+0000","last_became_peered":"2024-01-15T14:35:59.186991+0000","last_unstale":"2024-01-15T14:36:01.446418+0000","last_undegraded":"2024-01-15T14:36:01.446418+0000","last_fullsized":"2024-01-15T14:36:01.446418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:08.359295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178351+0000","last_change":"2024-01-15T14:35:58.715678+0000","last_active":"2024-01-15T14:36:00.178351+0000","last_peered":"2024-01-15T14:36:00.178351+0000","last_clean":"2024-01-15T14:36:00.178351+0000","last_became_active":"2024-01-15T14:35:58.715392+0000","last_became_peered":"2024-01-15T14:35:58.715392+0000","last_unstale":"2024-01-15T14:36:00.178351+0000","last_undegraded":"2024-01-15T14:36:00.178351+0000","last_fullsized":"2024-01-15T14:36:00.178351+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:45:53.496590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178265+0000","last_change":"2024-01-15T14:35:59.196451+0000","last_active":"2024-01-15T14:36:00.178265+0000","last_peered":"2024-01-15T14:36:00.178265+0000","last_clean":"2024-01-15T14:36:00.178265+0000","last_became_active":"2024-01-15T14:35:59.196237+0000","last_became_peered":"2024-01-15T14:35:59.196237+0000","last_unstale":"2024-01-15T14:36:00.178265+0000","last_undegraded":"2024-01-15T14:36:00.178265+0000","last_fullsized":"2024-01-15T14:36:00.178265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:01.066110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446468+0000","last_change":"2024-01-15T14:35:58.718612+0000","last_active":"2024-01-15T14:36:01.446468+0000","last_peered":"2024-01-15T14:36:01.446468+0000","last_clean":"2024-01-15T14:36:01.446468+0000","last_became_active":"2024-01-15T14:35:58.718350+0000","last_became_peered":"2024-01-15T14:35:58.718350+0000","last_unstale":"2024-01-15T14:36:01.446468+0000","last_undegraded":"2024-01-15T14:36:01.446468+0000","last_fullsized":"2024-01-15T14:36:01.446468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:29:50.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445464+0000","last_change":"2024-01-15T14:35:59.194990+0000","last_active":"2024-01-15T14:36:01.445464+0000","last_peered":"2024-01-15T14:36:01.445464+0000","last_clean":"2024-01-15T14:36:01.445464+0000","last_became_active":"2024-01-15T14:35:59.194701+0000","last_became_peered":"2024-01-15T14:35:59.194701+0000","last_unstale":"2024-01-15T14:36:01.445464+0000","last_undegraded":"2024-01-15T14:36:01.445464+0000","last_fullsized":"2024-01-15T14:36:01.445464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:18:13.350946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174596+0000","last_change":"2024-01-15T14:35:58.719422+0000","last_active":"2024-01-15T14:36:00.174596+0000","last_peered":"2024-01-15T14:36:00.174596+0000","last_clean":"2024-01-15T14:36:00.174596+0000","last_became_active":"2024-01-15T14:35:58.718448+0000","last_became_peered":"2024-01-15T14:35:58.718448+0000","last_unstale":"2024-01-15T14:36:00.174596+0000","last_undegraded":"2024-01-15T14:36:00.174596+0000","last_fullsized":"2024-01-15T14:36:00.174596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:46.766398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446870+0000","last_change":"2024-01-15T14:35:59.196067+0000","last_active":"2024-01-15T14:36:01.446870+0000","last_peered":"2024-01-15T14:36:01.446870+0000","last_clean":"2024-01-15T14:36:01.446870+0000","last_became_active":"2024-01-15T14:35:59.195874+0000","last_became_peered":"2024-01-15T14:35:59.195874+0000","last_unstale":"2024-01-15T14:36:01.446870+0000","last_undegraded":"2024-01-15T14:36:01.446870+0000","last_fullsized":"2024-01-15T14:36:01.446870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446318+0000","last_change":"2024-01-15T14:35:58.716192+0000","last_active":"2024-01-15T14:36:01.446318+0000","last_peered":"2024-01-15T14:36:01.446318+0000","last_clean":"2024-01-15T14:36:01.446318+0000","last_became_active":"2024-01-15T14:35:58.715905+0000","last_became_peered":"2024-01-15T14:35:58.715905+0000","last_unstale":"2024-01-15T14:36:01.446318+0000","last_undegraded":"2024-01-15T14:36:01.446318+0000","last_fullsized":"2024-01-15T14:36:01.446318+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:14.739333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177935+0000","last_change":"2024-01-15T14:35:59.197382+0000","last_active":"2024-01-15T14:36:00.177935+0000","last_peered":"2024-01-15T14:36:00.177935+0000","last_clean":"2024-01-15T14:36:00.177935+0000","last_became_active":"2024-01-15T14:35:59.197103+0000","last_became_peered":"2024-01-15T14:35:59.197103+0000","last_unstale":"2024-01-15T14:36:00.177935+0000","last_undegraded":"2024-01-15T14:36:00.177935+0000","last_fullsized":"2024-01-15T14:36:00.177935+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:30:42.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445397+0000","last_change":"2024-01-15T14:35:58.718597+0000","last_active":"2024-01-15T14:36:01.445397+0000","last_peered":"2024-01-15T14:36:01.445397+0000","last_clean":"2024-01-15T14:36:01.445397+0000","last_became_active":"2024-01-15T14:35:58.718339+0000","last_became_peered":"2024-01-15T14:35:58.718339+0000","last_unstale":"2024-01-15T14:36:01.445397+0000","last_undegraded":"2024-01-15T14:36:01.445397+0000","last_fullsized":"2024-01-15T14:36:01.445397+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:48.847777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345340+0000","last_change":"2024-01-15T14:35:59.188867+0000","last_active":"2024-01-15T14:36:01.345340+0000","last_peered":"2024-01-15T14:36:01.345340+0000","last_clean":"2024-01-15T14:36:01.345340+0000","last_became_active":"2024-01-15T14:35:59.186904+0000","last_became_peered":"2024-01-15T14:35:59.186904+0000","last_unstale":"2024-01-15T14:36:01.345340+0000","last_undegraded":"2024-01-15T14:36:01.345340+0000","last_fullsized":"2024-01-15T14:36:01.345340+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:04.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445238+0000","last_change":"2024-01-15T14:35:58.720981+0000","last_active":"2024-01-15T14:36:01.445238+0000","last_peered":"2024-01-15T14:36:01.445238+0000","last_clean":"2024-01-15T14:36:01.445238+0000","last_became_active":"2024-01-15T14:35:58.720645+0000","last_became_peered":"2024-01-15T14:35:58.720645+0000","last_unstale":"2024-01-15T14:36:01.445238+0000","last_undegraded":"2024-01-15T14:36:01.445238+0000","last_fullsized":"2024-01-15T14:36:01.445238+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:23:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445256+0000","last_change":"2024-01-15T14:35:59.199041+0000","last_active":"2024-01-15T14:36:01.445256+0000","last_peered":"2024-01-15T14:36:01.445256+0000","last_clean":"2024-01-15T14:36:01.445256+0000","last_became_active":"2024-01-15T14:35:59.198728+0000","last_became_peered":"2024-01-15T14:35:59.198728+0000","last_unstale":"2024-01-15T14:36:01.445256+0000","last_undegraded":"2024-01-15T14:36:01.445256+0000","last_fullsized":"2024-01-15T14:36:01.445256+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:02:35.153060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174081+0000","last_change":"2024-01-15T14:35:58.724180+0000","last_active":"2024-01-15T14:36:00.174081+0000","last_peered":"2024-01-15T14:36:00.174081+0000","last_clean":"2024-01-15T14:36:00.174081+0000","last_became_active":"2024-01-15T14:35:58.723873+0000","last_became_peered":"2024-01-15T14:35:58.723873+0000","last_unstale":"2024-01-15T14:36:00.174081+0000","last_undegraded":"2024-01-15T14:36:00.174081+0000","last_fullsized":"2024-01-15T14:36:00.174081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:00.057015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445805+0000","last_change":"2024-01-15T14:35:59.184327+0000","last_active":"2024-01-15T14:36:01.445805+0000","last_peered":"2024-01-15T14:36:01.445805+0000","last_clean":"2024-01-15T14:36:01.445805+0000","last_became_active":"2024-01-15T14:35:59.184112+0000","last_became_peered":"2024-01-15T14:35:59.184112+0000","last_unstale":"2024-01-15T14:36:01.445805+0000","last_undegraded":"2024-01-15T14:36:01.445805+0000","last_fullsized":"2024-01-15T14:36:01.445805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:47.775492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445144+0000","last_change":"2024-01-15T14:35:58.717974+0000","last_active":"2024-01-15T14:36:01.445144+0000","last_peered":"2024-01-15T14:36:01.445144+0000","last_clean":"2024-01-15T14:36:01.445144+0000","last_became_active":"2024-01-15T14:35:58.717679+0000","last_became_peered":"2024-01-15T14:35:58.717679+0000","last_unstale":"2024-01-15T14:36:01.445144+0000","last_undegraded":"2024-01-15T14:36:01.445144+0000","last_fullsized":"2024-01-15T14:36:01.445144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:40.876364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345413+0000","last_change":"2024-01-15T14:35:59.188560+0000","last_active":"2024-01-15T14:36:01.345413+0000","last_peered":"2024-01-15T14:36:01.345413+0000","last_clean":"2024-01-15T14:36:01.345413+0000","last_became_active":"2024-01-15T14:35:59.187878+0000","last_became_peered":"2024-01-15T14:35:59.187878+0000","last_unstale":"2024-01-15T14:36:01.345413+0000","last_undegraded":"2024-01-15T14:36:01.345413+0000","last_fullsized":"2024-01-15T14:36:01.345413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:40.243794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174372+0000","last_change":"2024-01-15T14:35:58.722504+0000","last_active":"2024-01-15T14:36:00.174372+0000","last_peered":"2024-01-15T14:36:00.174372+0000","last_clean":"2024-01-15T14:36:00.174372+0000","last_became_active":"2024-01-15T14:35:58.722227+0000","last_became_peered":"2024-01-15T14:35:58.722227+0000","last_unstale":"2024-01-15T14:36:00.174372+0000","last_undegraded":"2024-01-15T14:36:00.174372+0000","last_fullsized":"2024-01-15T14:36:00.174372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:31:41.196900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174525+0000","last_change":"2024-01-15T14:35:59.180752+0000","last_active":"2024-01-15T14:36:00.174525+0000","last_peered":"2024-01-15T14:36:00.174525+0000","last_clean":"2024-01-15T14:36:00.174525+0000","last_became_active":"2024-01-15T14:35:59.180504+0000","last_became_peered":"2024-01-15T14:35:59.180504+0000","last_unstale":"2024-01-15T14:36:00.174525+0000","last_undegraded":"2024-01-15T14:36:00.174525+0000","last_fullsized":"2024-01-15T14:36:00.174525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:21:21.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446502+0000","last_change":"2024-01-15T14:35:58.721258+0000","last_active":"2024-01-15T14:36:01.446502+0000","last_peered":"2024-01-15T14:36:01.446502+0000","last_clean":"2024-01-15T14:36:01.446502+0000","last_became_active":"2024-01-15T14:35:58.720974+0000","last_became_peered":"2024-01-15T14:35:58.720974+0000","last_unstale":"2024-01-15T14:36:01.446502+0000","last_undegraded":"2024-01-15T14:36:01.446502+0000","last_fullsized":"2024-01-15T14:36:01.446502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:23:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174760+0000","last_change":"2024-01-15T14:35:59.188093+0000","last_active":"2024-01-15T14:36:00.174760+0000","last_peered":"2024-01-15T14:36:00.174760+0000","last_clean":"2024-01-15T14:36:00.174760+0000","last_became_active":"2024-01-15T14:35:59.187810+0000","last_became_peered":"2024-01-15T14:35:59.187810+0000","last_unstale":"2024-01-15T14:36:00.174760+0000","last_undegraded":"2024-01-15T14:36:00.174760+0000","last_fullsized":"2024-01-15T14:36:00.174760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:59.382998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345317+0000","last_change":"2024-01-15T14:35:58.720133+0000","last_active":"2024-01-15T14:36:01.345317+0000","last_peered":"2024-01-15T14:36:01.345317+0000","last_clean":"2024-01-15T14:36:01.345317+0000","last_became_active":"2024-01-15T14:35:58.719855+0000","last_became_peered":"2024-01-15T14:35:58.719855+0000","last_unstale":"2024-01-15T14:36:01.345317+0000","last_undegraded":"2024-01-15T14:36:01.345317+0000","last_fullsized":"2024-01-15T14:36:01.345317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:29.126468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345358+0000","last_change":"2024-01-15T14:35:59.187707+0000","last_active":"2024-01-15T14:36:01.345358+0000","last_peered":"2024-01-15T14:36:01.345358+0000","last_clean":"2024-01-15T14:36:01.345358+0000","last_became_active":"2024-01-15T14:35:59.187081+0000","last_became_peered":"2024-01-15T14:35:59.187081+0000","last_unstale":"2024-01-15T14:36:01.345358+0000","last_undegraded":"2024-01-15T14:36:01.345358+0000","last_fullsized":"2024-01-15T14:36:01.345358+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:37.731830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174825+0000","last_change":"2024-01-15T14:35:58.719318+0000","last_active":"2024-01-15T14:36:00.174825+0000","last_peered":"2024-01-15T14:36:00.174825+0000","last_clean":"2024-01-15T14:36:00.174825+0000","last_became_active":"2024-01-15T14:35:58.719038+0000","last_became_peered":"2024-01-15T14:35:58.719038+0000","last_unstale":"2024-01-15T14:36:00.174825+0000","last_undegraded":"2024-01-15T14:36:00.174825+0000","last_fullsized":"2024-01-15T14:36:00.174825+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:43.351804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445921+0000","last_change":"2024-01-15T14:35:59.190039+0000","last_active":"2024-01-15T14:36:01.445921+0000","last_peered":"2024-01-15T14:36:01.445921+0000","last_clean":"2024-01-15T14:36:01.445921+0000","last_became_active":"2024-01-15T14:35:59.189775+0000","last_became_peered":"2024-01-15T14:35:59.189775+0000","last_unstale":"2024-01-15T14:36:01.445921+0000","last_undegraded":"2024-01-15T14:36:01.445921+0000","last_fullsized":"2024-01-15T14:36:01.445921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:19.359918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445513+0000","last_change":"2024-01-15T14:35:58.708011+0000","last_active":"2024-01-15T14:36:01.445513+0000","last_peered":"2024-01-15T14:36:01.445513+0000","last_clean":"2024-01-15T14:36:01.445513+0000","last_became_active":"2024-01-15T14:35:58.707767+0000","last_became_peered":"2024-01-15T14:35:58.707767+0000","last_unstale":"2024-01-15T14:36:01.445513+0000","last_undegraded":"2024-01-15T14:36:01.445513+0000","last_fullsized":"2024-01-15T14:36:01.445513+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:07.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174375+0000","last_change":"2024-01-15T14:35:59.182071+0000","last_active":"2024-01-15T14:36:00.174375+0000","last_peered":"2024-01-15T14:36:00.174375+0000","last_clean":"2024-01-15T14:36:00.174375+0000","last_became_active":"2024-01-15T14:35:59.181815+0000","last_became_peered":"2024-01-15T14:35:59.181815+0000","last_unstale":"2024-01-15T14:36:00.174375+0000","last_undegraded":"2024-01-15T14:36:00.174375+0000","last_fullsized":"2024-01-15T14:36:00.174375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:04.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446019+0000","last_change":"2024-01-15T14:35:58.720466+0000","last_active":"2024-01-15T14:36:01.446019+0000","last_peered":"2024-01-15T14:36:01.446019+0000","last_clean":"2024-01-15T14:36:01.446019+0000","last_became_active":"2024-01-15T14:35:58.720193+0000","last_became_peered":"2024-01-15T14:35:58.720193+0000","last_unstale":"2024-01-15T14:36:01.446019+0000","last_undegraded":"2024-01-15T14:36:01.446019+0000","last_fullsized":"2024-01-15T14:36:01.446019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:21.624021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177972+0000","last_change":"2024-01-15T14:35:59.197290+0000","last_active":"2024-01-15T14:36:00.177972+0000","last_peered":"2024-01-15T14:36:00.177972+0000","last_clean":"2024-01-15T14:36:00.177972+0000","last_became_active":"2024-01-15T14:35:59.197015+0000","last_became_peered":"2024-01-15T14:35:59.197015+0000","last_unstale":"2024-01-15T14:36:00.177972+0000","last_undegraded":"2024-01-15T14:36:00.177972+0000","last_fullsized":"2024-01-15T14:36:00.177972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174155+0000","last_change":"2024-01-15T14:35:58.722915+0000","last_active":"2024-01-15T14:36:00.174155+0000","last_peered":"2024-01-15T14:36:00.174155+0000","last_clean":"2024-01-15T14:36:00.174155+0000","last_became_active":"2024-01-15T14:35:58.722300+0000","last_became_peered":"2024-01-15T14:35:58.722300+0000","last_unstale":"2024-01-15T14:36:00.174155+0000","last_undegraded":"2024-01-15T14:36:00.174155+0000","last_fullsized":"2024-01-15T14:36:00.174155+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:26.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178145+0000","last_change":"2024-01-15T14:35:59.194805+0000","last_active":"2024-01-15T14:36:00.178145+0000","last_peered":"2024-01-15T14:36:00.178145+0000","last_clean":"2024-01-15T14:36:00.178145+0000","last_became_active":"2024-01-15T14:35:59.194545+0000","last_became_peered":"2024-01-15T14:35:59.194545+0000","last_unstale":"2024-01-15T14:36:00.178145+0000","last_undegraded":"2024-01-15T14:36:00.178145+0000","last_fullsized":"2024-01-15T14:36:00.178145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:27.154968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174032+0000","last_change":"2024-01-15T14:35:58.718692+0000","last_active":"2024-01-15T14:36:00.174032+0000","last_peered":"2024-01-15T14:36:00.174032+0000","last_clean":"2024-01-15T14:36:00.174032+0000","last_became_active":"2024-01-15T14:35:58.718433+0000","last_became_peered":"2024-01-15T14:35:58.718433+0000","last_unstale":"2024-01-15T14:36:00.174032+0000","last_undegraded":"2024-01-15T14:36:00.174032+0000","last_fullsized":"2024-01-15T14:36:00.174032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:21.191583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445652+0000","last_change":"2024-01-15T14:35:59.195055+0000","last_active":"2024-01-15T14:36:01.445652+0000","last_peered":"2024-01-15T14:36:01.445652+0000","last_clean":"2024-01-15T14:36:01.445652+0000","last_became_active":"2024-01-15T14:35:59.194468+0000","last_became_peered":"2024-01-15T14:35:59.194468+0000","last_unstale":"2024-01-15T14:36:01.445652+0000","last_undegraded":"2024-01-15T14:36:01.445652+0000","last_fullsized":"2024-01-15T14:36:01.445652+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446779+0000","last_change":"2024-01-15T14:35:58.722370+0000","last_active":"2024-01-15T14:36:01.446779+0000","last_peered":"2024-01-15T14:36:01.446779+0000","last_clean":"2024-01-15T14:36:01.446779+0000","last_became_active":"2024-01-15T14:35:58.722085+0000","last_became_peered":"2024-01-15T14:35:58.722085+0000","last_unstale":"2024-01-15T14:36:01.446779+0000","last_undegraded":"2024-01-15T14:36:01.446779+0000","last_fullsized":"2024-01-15T14:36:01.446779+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:31:41.196900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345435+0000","last_change":"2024-01-15T14:35:59.195889+0000","last_active":"2024-01-15T14:36:01.345435+0000","last_peered":"2024-01-15T14:36:01.345435+0000","last_clean":"2024-01-15T14:36:01.345435+0000","last_became_active":"2024-01-15T14:35:59.195574+0000","last_became_peered":"2024-01-15T14:35:59.195574+0000","last_unstale":"2024-01-15T14:36:01.345435+0000","last_undegraded":"2024-01-15T14:36:01.345435+0000","last_fullsized":"2024-01-15T14:36:01.345435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:21:21.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445615+0000","last_change":"2024-01-15T14:35:58.708707+0000","last_active":"2024-01-15T14:36:01.445615+0000","last_peered":"2024-01-15T14:36:01.445615+0000","last_clean":"2024-01-15T14:36:01.445615+0000","last_became_active":"2024-01-15T14:35:58.708419+0000","last_became_peered":"2024-01-15T14:35:58.708419+0000","last_unstale":"2024-01-15T14:36:01.445615+0000","last_undegraded":"2024-01-15T14:36:01.445615+0000","last_fullsized":"2024-01-15T14:36:01.445615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:26.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446511+0000","last_change":"2024-01-15T14:35:59.196118+0000","last_active":"2024-01-15T14:36:01.446511+0000","last_peered":"2024-01-15T14:36:01.446511+0000","last_clean":"2024-01-15T14:36:01.446511+0000","last_became_active":"2024-01-15T14:35:59.195858+0000","last_became_peered":"2024-01-15T14:35:59.195858+0000","last_unstale":"2024-01-15T14:36:01.446511+0000","last_undegraded":"2024-01-15T14:36:01.446511+0000","last_fullsized":"2024-01-15T14:36:01.446511+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445596+0000","last_change":"2024-01-15T14:35:58.724003+0000","last_active":"2024-01-15T14:36:01.445596+0000","last_peered":"2024-01-15T14:36:01.445596+0000","last_clean":"2024-01-15T14:36:01.445596+0000","last_became_active":"2024-01-15T14:35:58.723747+0000","last_became_peered":"2024-01-15T14:35:58.723747+0000","last_unstale":"2024-01-15T14:36:01.445596+0000","last_undegraded":"2024-01-15T14:36:01.445596+0000","last_fullsized":"2024-01-15T14:36:01.445596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:03.590388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446044+0000","last_change":"2024-01-15T14:35:59.191930+0000","last_active":"2024-01-15T14:36:01.446044+0000","last_peered":"2024-01-15T14:36:01.446044+0000","last_clean":"2024-01-15T14:36:01.446044+0000","last_became_active":"2024-01-15T14:35:59.191779+0000","last_became_peered":"2024-01-15T14:35:59.191779+0000","last_unstale":"2024-01-15T14:36:01.446044+0000","last_undegraded":"2024-01-15T14:36:01.446044+0000","last_fullsized":"2024-01-15T14:36:01.446044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:29:51.441617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445531+0000","last_change":"2024-01-15T14:35:58.716285+0000","last_active":"2024-01-15T14:36:01.445531+0000","last_peered":"2024-01-15T14:36:01.445531+0000","last_clean":"2024-01-15T14:36:01.445531+0000","last_became_active":"2024-01-15T14:35:58.716016+0000","last_became_peered":"2024-01-15T14:35:58.716016+0000","last_unstale":"2024-01-15T14:36:01.445531+0000","last_undegraded":"2024-01-15T14:36:01.445531+0000","last_fullsized":"2024-01-15T14:36:01.445531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:21.624021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446762+0000","last_change":"2024-01-15T14:35:59.197943+0000","last_active":"2024-01-15T14:36:01.446762+0000","last_peered":"2024-01-15T14:36:01.446762+0000","last_clean":"2024-01-15T14:36:01.446762+0000","last_became_active":"2024-01-15T14:35:59.197679+0000","last_became_peered":"2024-01-15T14:35:59.197679+0000","last_unstale":"2024-01-15T14:36:01.446762+0000","last_undegraded":"2024-01-15T14:36:01.446762+0000","last_fullsized":"2024-01-15T14:36:01.446762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:27.154968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446107+0000","last_change":"2024-01-15T14:35:58.719317+0000","last_active":"2024-01-15T14:36:01.446107+0000","last_peered":"2024-01-15T14:36:01.446107+0000","last_clean":"2024-01-15T14:36:01.446107+0000","last_became_active":"2024-01-15T14:35:58.719069+0000","last_became_peered":"2024-01-15T14:35:58.719069+0000","last_unstale":"2024-01-15T14:36:01.446107+0000","last_undegraded":"2024-01-15T14:36:01.446107+0000","last_fullsized":"2024-01-15T14:36:01.446107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:00.057015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446037+0000","last_change":"2024-01-15T14:35:59.190777+0000","last_active":"2024-01-15T14:36:01.446037+0000","last_peered":"2024-01-15T14:36:01.446037+0000","last_clean":"2024-01-15T14:36:01.446037+0000","last_became_active":"2024-01-15T14:35:59.190526+0000","last_became_peered":"2024-01-15T14:35:59.190526+0000","last_unstale":"2024-01-15T14:36:01.446037+0000","last_undegraded":"2024-01-15T14:36:01.446037+0000","last_fullsized":"2024-01-15T14:36:01.446037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:45:54.505684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178709+0000","last_change":"2024-01-15T14:35:58.715685+0000","last_active":"2024-01-15T14:36:00.178709+0000","last_peered":"2024-01-15T14:36:00.178709+0000","last_clean":"2024-01-15T14:36:00.178709+0000","last_became_active":"2024-01-15T14:35:58.715406+0000","last_became_peered":"2024-01-15T14:35:58.715406+0000","last_unstale":"2024-01-15T14:36:00.178709+0000","last_undegraded":"2024-01-15T14:36:00.178709+0000","last_fullsized":"2024-01-15T14:36:00.178709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:26.118609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445469+0000","last_change":"2024-01-15T14:35:59.190741+0000","last_active":"2024-01-15T14:36:01.445469+0000","last_peered":"2024-01-15T14:36:01.445469+0000","last_clean":"2024-01-15T14:36:01.445469+0000","last_became_active":"2024-01-15T14:35:59.190474+0000","last_became_peered":"2024-01-15T14:35:59.190474+0000","last_unstale":"2024-01-15T14:36:01.445469+0000","last_undegraded":"2024-01-15T14:36:01.445469+0000","last_fullsized":"2024-01-15T14:36:01.445469+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:35:47.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445241+0000","last_change":"2024-01-15T14:35:58.715057+0000","last_active":"2024-01-15T14:36:01.445241+0000","last_peered":"2024-01-15T14:36:01.445241+0000","last_clean":"2024-01-15T14:36:01.445241+0000","last_became_active":"2024-01-15T14:35:58.714781+0000","last_became_peered":"2024-01-15T14:35:58.714781+0000","last_unstale":"2024-01-15T14:36:01.445241+0000","last_undegraded":"2024-01-15T14:36:01.445241+0000","last_fullsized":"2024-01-15T14:36:01.445241+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:21.191583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178728+0000","last_change":"2024-01-15T14:35:59.192344+0000","last_active":"2024-01-15T14:36:00.178728+0000","last_peered":"2024-01-15T14:36:00.178728+0000","last_clean":"2024-01-15T14:36:00.178728+0000","last_became_active":"2024-01-15T14:35:59.191891+0000","last_became_peered":"2024-01-15T14:35:59.191891+0000","last_unstale":"2024-01-15T14:36:00.178728+0000","last_undegraded":"2024-01-15T14:36:00.178728+0000","last_fullsized":"2024-01-15T14:36:00.178728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:21:21.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.344911+0000","last_change":"2024-01-15T14:35:58.716350+0000","last_active":"2024-01-15T14:36:01.344911+0000","last_peered":"2024-01-15T14:36:01.344911+0000","last_clean":"2024-01-15T14:36:01.344911+0000","last_became_active":"2024-01-15T14:35:58.715634+0000","last_became_peered":"2024-01-15T14:35:58.715634+0000","last_unstale":"2024-01-15T14:36:01.344911+0000","last_undegraded":"2024-01-15T14:36:01.344911+0000","last_fullsized":"2024-01-15T14:36:01.344911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:18.350823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446640+0000","last_change":"2024-01-15T14:35:59.189401+0000","last_active":"2024-01-15T14:36:01.446640+0000","last_peered":"2024-01-15T14:36:01.446640+0000","last_clean":"2024-01-15T14:36:01.446640+0000","last_became_active":"2024-01-15T14:35:59.188947+0000","last_became_peered":"2024-01-15T14:35:59.188947+0000","last_unstale":"2024-01-15T14:36:01.446640+0000","last_undegraded":"2024-01-15T14:36:01.446640+0000","last_fullsized":"2024-01-15T14:36:01.446640+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:22.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446626+0000","last_change":"2024-01-15T14:35:58.720184+0000","last_active":"2024-01-15T14:36:01.446626+0000","last_peered":"2024-01-15T14:36:01.446626+0000","last_clean":"2024-01-15T14:36:01.446626+0000","last_became_active":"2024-01-15T14:35:58.719875+0000","last_became_peered":"2024-01-15T14:35:58.719875+0000","last_unstale":"2024-01-15T14:36:01.446626+0000","last_undegraded":"2024-01-15T14:36:01.446626+0000","last_fullsized":"2024-01-15T14:36:01.446626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:07.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345544+0000","last_change":"2024-01-15T14:35:59.188314+0000","last_active":"2024-01-15T14:36:01.345544+0000","last_peered":"2024-01-15T14:36:01.345544+0000","last_clean":"2024-01-15T14:36:01.345544+0000","last_became_active":"2024-01-15T14:35:59.186735+0000","last_became_peered":"2024-01-15T14:35:59.186735+0000","last_unstale":"2024-01-15T14:36:01.345544+0000","last_undegraded":"2024-01-15T14:36:01.345544+0000","last_fullsized":"2024-01-15T14:36:01.345544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:00.462542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.738774+0000","last_change":"2024-01-15T14:35:58.719333+0000","last_active":"2024-01-15T14:36:01.738774+0000","last_peered":"2024-01-15T14:36:01.738774+0000","last_clean":"2024-01-15T14:36:01.738774+0000","last_became_active":"2024-01-15T14:35:58.719079+0000","last_became_peered":"2024-01-15T14:35:58.719079+0000","last_unstale":"2024-01-15T14:36:01.738774+0000","last_undegraded":"2024-01-15T14:36:01.738774+0000","last_fullsized":"2024-01-15T14:36:01.738774+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:21.191583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445755+0000","last_change":"2024-01-15T14:35:59.186650+0000","last_active":"2024-01-15T14:36:01.445755+0000","last_peered":"2024-01-15T14:36:01.445755+0000","last_clean":"2024-01-15T14:36:01.445755+0000","last_became_active":"2024-01-15T14:35:59.186396+0000","last_became_peered":"2024-01-15T14:35:59.186396+0000","last_unstale":"2024-01-15T14:36:01.445755+0000","last_undegraded":"2024-01-15T14:36:01.445755+0000","last_fullsized":"2024-01-15T14:36:01.445755+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:22.633116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446269+0000","last_change":"2024-01-15T14:35:58.724022+0000","last_active":"2024-01-15T14:36:01.446269+0000","last_peered":"2024-01-15T14:36:01.446269+0000","last_clean":"2024-01-15T14:36:01.446269+0000","last_became_active":"2024-01-15T14:35:58.723742+0000","last_became_peered":"2024-01-15T14:35:58.723742+0000","last_unstale":"2024-01-15T14:36:01.446269+0000","last_undegraded":"2024-01-15T14:36:01.446269+0000","last_fullsized":"2024-01-15T14:36:01.446269+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:40.876364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178619+0000","last_change":"2024-01-15T14:35:59.188111+0000","last_active":"2024-01-15T14:36:00.178619+0000","last_peered":"2024-01-15T14:36:00.178619+0000","last_clean":"2024-01-15T14:36:00.178619+0000","last_became_active":"2024-01-15T14:35:59.187805+0000","last_became_peered":"2024-01-15T14:35:59.187805+0000","last_unstale":"2024-01-15T14:36:00.178619+0000","last_undegraded":"2024-01-15T14:36:00.178619+0000","last_fullsized":"2024-01-15T14:36:00.178619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:49.856872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446015+0000","last_change":"2024-01-15T14:35:58.715522+0000","last_active":"2024-01-15T14:36:01.446015+0000","last_peered":"2024-01-15T14:36:01.446015+0000","last_clean":"2024-01-15T14:36:01.446015+0000","last_became_active":"2024-01-15T14:35:58.715173+0000","last_became_peered":"2024-01-15T14:35:58.715173+0000","last_unstale":"2024-01-15T14:36:01.446015+0000","last_undegraded":"2024-01-15T14:36:01.446015+0000","last_fullsized":"2024-01-15T14:36:01.446015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:26.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446347+0000","last_change":"2024-01-15T14:35:59.186736+0000","last_active":"2024-01-15T14:36:01.446347+0000","last_peered":"2024-01-15T14:36:01.446347+0000","last_clean":"2024-01-15T14:36:01.446347+0000","last_became_active":"2024-01-15T14:35:59.186445+0000","last_became_peered":"2024-01-15T14:35:59.186445+0000","last_unstale":"2024-01-15T14:36:01.446347+0000","last_undegraded":"2024-01-15T14:36:01.446347+0000","last_fullsized":"2024-01-15T14:36:01.446347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:41.885459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446717+0000","last_change":"2024-01-15T14:35:58.717510+0000","last_active":"2024-01-15T14:36:01.446717+0000","last_peered":"2024-01-15T14:36:01.446717+0000","last_clean":"2024-01-15T14:36:01.446717+0000","last_became_active":"2024-01-15T14:35:58.717199+0000","last_became_peered":"2024-01-15T14:35:58.717199+0000","last_unstale":"2024-01-15T14:36:01.446717+0000","last_undegraded":"2024-01-15T14:36:01.446717+0000","last_fullsized":"2024-01-15T14:36:01.446717+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:48.847777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174086+0000","last_change":"2024-01-15T14:35:59.181030+0000","last_active":"2024-01-15T14:36:00.174086+0000","last_peered":"2024-01-15T14:36:00.174086+0000","last_clean":"2024-01-15T14:36:00.174086+0000","last_became_active":"2024-01-15T14:35:59.180778+0000","last_became_peered":"2024-01-15T14:35:59.180778+0000","last_unstale":"2024-01-15T14:36:00.174086+0000","last_undegraded":"2024-01-15T14:36:00.174086+0000","last_fullsized":"2024-01-15T14:36:00.174086+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445317+0000","last_change":"2024-01-15T14:35:58.711993+0000","last_active":"2024-01-15T14:36:01.445317+0000","last_peered":"2024-01-15T14:36:01.445317+0000","last_clean":"2024-01-15T14:36:01.445317+0000","last_became_active":"2024-01-15T14:35:58.711721+0000","last_became_peered":"2024-01-15T14:35:58.711721+0000","last_unstale":"2024-01-15T14:36:01.445317+0000","last_undegraded":"2024-01-15T14:36:01.445317+0000","last_fullsized":"2024-01-15T14:36:01.445317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:00.057015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445172+0000","last_change":"2024-01-15T14:35:59.195866+0000","last_active":"2024-01-15T14:36:01.445172+0000","last_peered":"2024-01-15T14:36:01.445172+0000","last_clean":"2024-01-15T14:36:01.445172+0000","last_became_active":"2024-01-15T14:35:59.195567+0000","last_became_peered":"2024-01-15T14:35:59.195567+0000","last_unstale":"2024-01-15T14:36:01.445172+0000","last_undegraded":"2024-01-15T14:36:01.445172+0000","last_fullsized":"2024-01-15T14:36:01.445172+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:31:42.205995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174177+0000","last_change":"2024-01-15T14:35:58.722668+0000","last_active":"2024-01-15T14:36:00.174177+0000","last_peered":"2024-01-15T14:36:00.174177+0000","last_clean":"2024-01-15T14:36:00.174177+0000","last_became_active":"2024-01-15T14:35:58.722367+0000","last_became_peered":"2024-01-15T14:35:58.722367+0000","last_unstale":"2024-01-15T14:36:00.174177+0000","last_undegraded":"2024-01-15T14:36:00.174177+0000","last_fullsized":"2024-01-15T14:36:00.174177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:03.590388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178221+0000","last_change":"2024-01-15T14:35:59.191746+0000","last_active":"2024-01-15T14:36:00.178221+0000","last_peered":"2024-01-15T14:36:00.178221+0000","last_clean":"2024-01-15T14:36:00.178221+0000","last_became_active":"2024-01-15T14:35:59.191440+0000","last_became_peered":"2024-01-15T14:35:59.191440+0000","last_unstale":"2024-01-15T14:36:00.178221+0000","last_undegraded":"2024-01-15T14:36:00.178221+0000","last_fullsized":"2024-01-15T14:36:00.178221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:04.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177843+0000","last_change":"2024-01-15T14:35:58.716125+0000","last_active":"2024-01-15T14:36:00.177843+0000","last_peered":"2024-01-15T14:36:00.177843+0000","last_clean":"2024-01-15T14:36:00.177843+0000","last_became_active":"2024-01-15T14:35:58.715847+0000","last_became_peered":"2024-01-15T14:35:58.715847+0000","last_unstale":"2024-01-15T14:36:00.177843+0000","last_undegraded":"2024-01-15T14:36:00.177843+0000","last_fullsized":"2024-01-15T14:36:00.177843+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:14.739333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T14:35:59.181837+0000","last_change":"2024-01-15T14:34:54.738502+0000","last_active":"2024-01-15T14:35:59.181837+0000","last_peered":"2024-01-15T14:35:59.181837+0000","last_clean":"2024-01-15T14:35:59.181837+0000","last_became_active":"2024-01-15T14:34:54.737354+0000","last_became_peered":"2024-01-15T14:34:54.737354+0000","last_unstale":"2024-01-15T14:35:59.181837+0000","last_undegraded":"2024-01-15T14:35:59.181837+0000","last_fullsized":"2024-01-15T14:35:59.181837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_clean_scrub_stamp":"2024-01-15T14:34:53.124419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:09:14.038173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":741713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.71799999999999997}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.626},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.60499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":151871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.69699999999999995}]},{"osd":2,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.60099999999999998}]},{"osd":3,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57999999999999996}]},{"osd":4,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.38},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":146589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.22}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.308}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.625}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.53500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":741623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.3369999999999997,"5min":6.3369999999999997,"15min":6.3369999999999997},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.0339999999999998,"5min":4.0339999999999998,"15min":4.0339999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.4669999999999996,"5min":6.4669999999999996,"15min":6.4669999999999996},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":19.106000000000002,"5min":19.106000000000002,"15min":19.106000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":6.383,"5min":6.383,"15min":6.383},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":19.045999999999999,"5min":19.045999999999999,"15min":19.045999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.599,"15min":1.599},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":18.984999999999999,"5min":18.984999999999999,"15min":18.984999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":4.0650000000000004,"5min":4.0650000000000004,"15min":4.0650000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.4390000000000001,"5min":6.4390000000000001,"15min":6.4390000000000001},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":6.4119999999999999,"5min":6.4119999999999999,"15min":6.4119999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":19.079000000000001,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.308,"15min":0.308},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34000000000000002}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41199999999999998}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.53400000000000003}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.5}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":152889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"last":0.30499999999999999}]},{"osd":1,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.495},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.45600000000000002}]},{"osd":6,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.442}]},{"osd":7,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44700000000000001}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.5},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.5869999999999997,"5min":4.5869999999999997,"15min":4.5869999999999997},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.65000000000000002}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.622},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.629},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.69999999999999996}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.0819999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.1899999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.77100000000000002}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":7.8230000000000004,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.871}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T14:36:03.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T14:36:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.016+0000 7f6e0c017700 1 Processor -- start 2024-01-15T14:36:04.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.016+0000 7f6e0c017700 1 -- start start 2024-01-15T14:36:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.017+0000 7f6e0c017700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 0x7f6e041603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.017+0000 7f6e0c017700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040cd100 con 0x7f6e040d2160 2024-01-15T14:36:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.017+0000 7f6e0c017700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040d09a0 con 0x7f6e0415bed0 2024-01-15T14:36:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.017+0000 7f6e0c017700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b1ea0 con 0x7f6e040b19f0 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.017+0000 7f6e095b2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6e040d2160 0x7f6e040cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54218/0 (socket says 172.21.15.77:54218) 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.017+0000 7f6e095b2700 1 -- 172.21.15.77:0/624653634 learned_addr learned my addr 172.21.15.77:0/624653634 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871848071 0 0) 0x7f6e040cd100 con 0x7f6e040d2160 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df00035b0 con 0x7f6e040d2160 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e0a5b4700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 0x7f6e041603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074103288 0 0) 0x7f6df00035b0 con 0x7f6e040d2160 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 msgr2=0x7f6e041603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 0x7f6e041603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 >> v1:172.21.15.159:6789/0 conn(0x7f6e0415bed0 legacy=0x7f6e040d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040cc7d0 con 0x7f6e040d2160 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f6e00001670 con 0x7f6e040d2160 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 >> v1:172.21.15.77:6789/0 conn(0x7f6e040d2160 legacy=0x7f6e040cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0003a30 0x7f6df0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6df0004850 0x7f6df0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 0x7f6df0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b1ea0 con 0x7f6df0004850 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df000bac0 con 0x7f6df0003a30 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df000bc80 con 0x7f6e040b19f0 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e095b2700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6df0004850 0x7f6df0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e09db3700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0003a30 0x7f6df0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e0a5b4700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 0x7f6df0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e095b2700 1 -- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 msgr2=0x7f6df0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e095b2700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e040b19f0 0x7f6df0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e095b2700 1 -- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0003a30 msgr2=0x7f6df0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e095b2700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0003a30 0x7f6df0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.018+0000 7f6e095b2700 1 -- 172.21.15.77:0/624653634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00035b0 con 0x7f6df0004850 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e095b2700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6df0004850 0x7f6df0004cd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6e00001eb0 tx=0x7f6df0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed81f80c299e8a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000cd50 con 0x7f6df0004850 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e00005960 con 0x7f6df0004850 2024-01-15T14:36:04.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e08db1700 1 -- 172.21.15.77:0/624653634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00005d10 con 0x7f6df0004850 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0c017700 1 -- 172.21.15.77:0/624653634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6df0004850 msgr2=0x7f6df0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0c017700 1 --2- 172.21.15.77:0/624653634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6df0004850 0x7f6df0004cd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6e00001eb0 tx=0x7f6df0005880 comp rx=0 tx=0).stop 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0a5b4700 1 -- 172.21.15.77:0/624653634 reap_dead start 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0c017700 1 -- 172.21.15.77:0/624653634 shutdown_connections 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0c017700 1 -- 172.21.15.77:0/624653634 >> 172.21.15.77:0/624653634 conn(0x7f6e04073130 msgr2=0x7f6e0409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0c017700 1 -- 172.21.15.77:0/624653634 shutdown_connections 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.019+0000 7f6e0c017700 1 -- 172.21.15.77:0/624653634 wait complete. 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 Processor -- start 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 -- start start 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0004850 0x7f6e040af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e0414fb50 0x7f6e0414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 0x7f6e040c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b4a50 con 0x7f6e040c5780 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040c6140 con 0x7f6df0004850 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0c017700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e041503d0 con 0x7f6e0414fb50 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0a5b4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 0x7f6e040c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.021+0000 7f6e0a5b4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 0x7f6e040c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57584/0 (socket says 172.21.15.77:57584) 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 -- 172.21.15.77:0/1766982067 learned_addr learned my addr 172.21.15.77:0/1766982067 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e09db3700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0004850 0x7f6e040af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e095b2700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e0414fb50 0x7f6e0414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 -- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e0414fb50 msgr2=0x7f6e0414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e0414fb50 0x7f6e0414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 -- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0004850 msgr2=0x7f6e040af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0004850 0x7f6e040af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 -- 172.21.15.77:0/1766982067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040b0740 con 0x7f6e040c5780 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0a5b4700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 0x7f6e040c5c00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6e040cf700 tx=0x7f6dfc00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00e040 con 0x7f6e040c5780 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc00f040 con 0x7f6e040c5780 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e040afc20 con 0x7f6e040c5780 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc015670 con 0x7f6e040c5780 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.022+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e040afea0 con 0x7f6e040c5780 2024-01-15T14:36:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.023+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e040b4b90 con 0x7f6e040c5780 2024-01-15T14:36:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.024+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6dfc0157d0 con 0x7f6e040c5780 2024-01-15T14:36:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.024+0000 7f6dfa7fc700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6de806b350 0x7f6de806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.024+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dfc08a480 con 0x7f6e040c5780 2024-01-15T14:36:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.024+0000 7f6e09db3700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6de806b350 0x7f6de806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.024+0000 7f6e09db3700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6de806b350 0x7f6de806d810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6df4007fd0 tx=0x7f6df4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.028+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e040b4b90 con 0x7f6e040c5780 2024-01-15T14:36:04.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.217+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e040ccd90 con 0x7f6de806b350 2024-01-15T14:36:04.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.225+0000 7f6dfa7fc700 1 -- 172.21.15.77:0/1766982067 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289253 (secure 0 0 0) 0x7f6df40012f0 con 0x7f6de806b350 2024-01-15T14:36:04.231 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:04.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6de806b350 msgr2=0x7f6de806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6de806b350 0x7f6de806d810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6df4007fd0 tx=0x7f6df4009040 comp rx=0 tx=0).stop 2024-01-15T14:36:04.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 msgr2=0x7f6e040c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 0x7f6e040c5c00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6e040cf700 tx=0x7f6dfc00f490 comp rx=0 tx=0).stop 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 shutdown_connections 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6de806b350 0x7f6de806d810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e040c5780 0x7f6e040c5c00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6df0004850 0x7f6e040af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 --2- 172.21.15.77:0/1766982067 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6e0414fb50 0x7f6e0414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 >> 172.21.15.77:0/1766982067 conn(0x7f6e04073130 msgr2=0x7f6e04159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 shutdown_connections 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.229+0000 7f6e0c017700 1 -- 172.21.15.77:0/1766982067 wait complete. 2024-01-15T14:36:04.235 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-01-15T14:36:04.242 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-01-15T14:36:03.031062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220392,"kb_used_data":4208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754328,"statfs":{"total":773094113280,"available":772868431872,"internally_reserved":0,"allocated":4308992,"data_stored":2382996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331426"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178124+0000","last_change":"2024-01-15T14:35:59.191318+0000","last_active":"2024-01-15T14:36:00.178124+0000","last_peered":"2024-01-15T14:36:00.178124+0000","last_clean":"2024-01-15T14:36:00.178124+0000","last_became_active":"2024-01-15T14:35:59.191035+0000","last_became_peered":"2024-01-15T14:35:59.191035+0000","last_unstale":"2024-01-15T14:36:00.178124+0000","last_undegraded":"2024-01-15T14:36:00.178124+0000","last_fullsized":"2024-01-15T14:36:00.178124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:22.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174537+0000","last_change":"2024-01-15T14:35:59.184745+0000","last_active":"2024-01-15T14:36:00.174537+0000","last_peered":"2024-01-15T14:36:00.174537+0000","last_clean":"2024-01-15T14:36:00.174537+0000","last_became_active":"2024-01-15T14:35:59.184484+0000","last_became_peered":"2024-01-15T14:35:59.184484+0000","last_unstale":"2024-01-15T14:36:00.174537+0000","last_undegraded":"2024-01-15T14:36:00.174537+0000","last_fullsized":"2024-01-15T14:36:00.174537+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:00.462542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345194+0000","last_change":"2024-01-15T14:35:59.189023+0000","last_active":"2024-01-15T14:36:01.345194+0000","last_peered":"2024-01-15T14:36:01.345194+0000","last_clean":"2024-01-15T14:36:01.345194+0000","last_became_active":"2024-01-15T14:35:59.187427+0000","last_became_peered":"2024-01-15T14:35:59.187427+0000","last_unstale":"2024-01-15T14:36:01.345194+0000","last_undegraded":"2024-01-15T14:36:01.345194+0000","last_fullsized":"2024-01-15T14:36:01.345194+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:30:42.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446328+0000","last_change":"2024-01-15T14:35:59.194817+0000","last_active":"2024-01-15T14:36:01.446328+0000","last_peered":"2024-01-15T14:36:01.446328+0000","last_clean":"2024-01-15T14:36:01.446328+0000","last_became_active":"2024-01-15T14:35:59.194526+0000","last_became_peered":"2024-01-15T14:35:59.194526+0000","last_unstale":"2024-01-15T14:36:01.446328+0000","last_undegraded":"2024-01-15T14:36:01.446328+0000","last_fullsized":"2024-01-15T14:36:01.446328+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:30.135562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445860+0000","last_change":"2024-01-15T14:35:59.187821+0000","last_active":"2024-01-15T14:36:01.445860+0000","last_peered":"2024-01-15T14:36:01.445860+0000","last_clean":"2024-01-15T14:36:01.445860+0000","last_became_active":"2024-01-15T14:35:59.187568+0000","last_became_peered":"2024-01-15T14:35:59.187568+0000","last_unstale":"2024-01-15T14:36:01.445860+0000","last_undegraded":"2024-01-15T14:36:01.445860+0000","last_fullsized":"2024-01-15T14:36:01.445860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:33:57.617411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174422+0000","last_change":"2024-01-15T14:35:59.189045+0000","last_active":"2024-01-15T14:36:00.174422+0000","last_peered":"2024-01-15T14:36:00.174422+0000","last_clean":"2024-01-15T14:36:00.174422+0000","last_became_active":"2024-01-15T14:35:59.188788+0000","last_became_peered":"2024-01-15T14:35:59.188788+0000","last_unstale":"2024-01-15T14:36:00.174422+0000","last_undegraded":"2024-01-15T14:36:00.174422+0000","last_fullsized":"2024-01-15T14:36:00.174422+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:40.243794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178585+0000","last_change":"2024-01-15T14:35:59.192905+0000","last_active":"2024-01-15T14:36:00.178585+0000","last_peered":"2024-01-15T14:36:00.178585+0000","last_clean":"2024-01-15T14:36:00.178585+0000","last_became_active":"2024-01-15T14:35:59.192639+0000","last_became_peered":"2024-01-15T14:35:59.192639+0000","last_unstale":"2024-01-15T14:36:00.178585+0000","last_undegraded":"2024-01-15T14:36:00.178585+0000","last_fullsized":"2024-01-15T14:36:00.178585+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:40.243794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177530+0000","last_change":"2024-01-15T14:35:59.191333+0000","last_active":"2024-01-15T14:36:00.177530+0000","last_peered":"2024-01-15T14:36:00.177530+0000","last_clean":"2024-01-15T14:36:00.177530+0000","last_became_active":"2024-01-15T14:35:59.191031+0000","last_became_peered":"2024-01-15T14:35:59.191031+0000","last_unstale":"2024-01-15T14:36:00.177530+0000","last_undegraded":"2024-01-15T14:36:00.177530+0000","last_fullsized":"2024-01-15T14:36:00.177530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:30.135562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445740+0000","last_change":"2024-01-15T14:35:59.198480+0000","last_active":"2024-01-15T14:36:01.445740+0000","last_peered":"2024-01-15T14:36:01.445740+0000","last_clean":"2024-01-15T14:36:01.445740+0000","last_became_active":"2024-01-15T14:35:59.197899+0000","last_became_peered":"2024-01-15T14:35:59.197899+0000","last_unstale":"2024-01-15T14:36:01.445740+0000","last_undegraded":"2024-01-15T14:36:01.445740+0000","last_fullsized":"2024-01-15T14:36:01.445740+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446451+0000","last_change":"2024-01-15T14:35:59.186140+0000","last_active":"2024-01-15T14:36:01.446451+0000","last_peered":"2024-01-15T14:36:01.446451+0000","last_clean":"2024-01-15T14:36:01.446451+0000","last_became_active":"2024-01-15T14:35:59.185835+0000","last_became_peered":"2024-01-15T14:35:59.185835+0000","last_unstale":"2024-01-15T14:36:01.446451+0000","last_undegraded":"2024-01-15T14:36:01.446451+0000","last_fullsized":"2024-01-15T14:36:01.446451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:18:13.350946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177931+0000","last_change":"2024-01-15T14:35:59.191736+0000","last_active":"2024-01-15T14:36:00.177931+0000","last_peered":"2024-01-15T14:36:00.177931+0000","last_clean":"2024-01-15T14:36:00.177931+0000","last_became_active":"2024-01-15T14:35:59.191407+0000","last_became_peered":"2024-01-15T14:35:59.191407+0000","last_unstale":"2024-01-15T14:36:00.177931+0000","last_undegraded":"2024-01-15T14:36:00.177931+0000","last_fullsized":"2024-01-15T14:36:00.177931+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345184+0000","last_change":"2024-01-15T14:35:59.188228+0000","last_active":"2024-01-15T14:36:01.345184+0000","last_peered":"2024-01-15T14:36:01.345184+0000","last_clean":"2024-01-15T14:36:01.345184+0000","last_became_active":"2024-01-15T14:35:59.187911+0000","last_became_peered":"2024-01-15T14:35:59.187911+0000","last_unstale":"2024-01-15T14:36:01.345184+0000","last_undegraded":"2024-01-15T14:36:01.345184+0000","last_fullsized":"2024-01-15T14:36:01.345184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446112+0000","last_change":"2024-01-15T14:35:59.190235+0000","last_active":"2024-01-15T14:36:01.446112+0000","last_peered":"2024-01-15T14:36:01.446112+0000","last_clean":"2024-01-15T14:36:01.446112+0000","last_became_active":"2024-01-15T14:35:59.189962+0000","last_became_peered":"2024-01-15T14:35:59.189962+0000","last_unstale":"2024-01-15T14:36:01.446112+0000","last_undegraded":"2024-01-15T14:36:01.446112+0000","last_fullsized":"2024-01-15T14:36:01.446112+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:23:33.956747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174377+0000","last_change":"2024-01-15T14:35:59.188450+0000","last_active":"2024-01-15T14:36:00.174377+0000","last_peered":"2024-01-15T14:36:00.174377+0000","last_clean":"2024-01-15T14:36:00.174377+0000","last_became_active":"2024-01-15T14:35:59.188178+0000","last_became_peered":"2024-01-15T14:35:59.188178+0000","last_unstale":"2024-01-15T14:36:00.174377+0000","last_undegraded":"2024-01-15T14:36:00.174377+0000","last_fullsized":"2024-01-15T14:36:00.174377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:37.731830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345173+0000","last_change":"2024-01-15T14:35:59.189385+0000","last_active":"2024-01-15T14:36:01.345173+0000","last_peered":"2024-01-15T14:36:01.345173+0000","last_clean":"2024-01-15T14:36:01.345173+0000","last_became_active":"2024-01-15T14:35:59.189168+0000","last_became_peered":"2024-01-15T14:35:59.189168+0000","last_unstale":"2024-01-15T14:36:01.345173+0000","last_undegraded":"2024-01-15T14:36:01.345173+0000","last_fullsized":"2024-01-15T14:36:01.345173+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177752+0000","last_change":"2024-01-15T14:35:59.190674+0000","last_active":"2024-01-15T14:36:00.177752+0000","last_peered":"2024-01-15T14:36:00.177752+0000","last_clean":"2024-01-15T14:36:00.177752+0000","last_became_active":"2024-01-15T14:35:59.190405+0000","last_became_peered":"2024-01-15T14:35:59.190405+0000","last_unstale":"2024-01-15T14:36:00.177752+0000","last_undegraded":"2024-01-15T14:36:00.177752+0000","last_fullsized":"2024-01-15T14:36:00.177752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:18:13.350946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178430+0000","last_change":"2024-01-15T14:35:59.192979+0000","last_active":"2024-01-15T14:36:00.178430+0000","last_peered":"2024-01-15T14:36:00.178430+0000","last_clean":"2024-01-15T14:36:00.178430+0000","last_became_active":"2024-01-15T14:35:59.192705+0000","last_became_peered":"2024-01-15T14:35:59.192705+0000","last_unstale":"2024-01-15T14:36:00.178430+0000","last_undegraded":"2024-01-15T14:36:00.178430+0000","last_fullsized":"2024-01-15T14:36:00.178430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:37.731830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174645+0000","last_change":"2024-01-15T14:35:59.181560+0000","last_active":"2024-01-15T14:36:00.174645+0000","last_peered":"2024-01-15T14:36:00.174645+0000","last_clean":"2024-01-15T14:36:00.174645+0000","last_became_active":"2024-01-15T14:35:59.181308+0000","last_became_peered":"2024-01-15T14:35:59.181308+0000","last_unstale":"2024-01-15T14:36:00.174645+0000","last_undegraded":"2024-01-15T14:36:00.174645+0000","last_fullsized":"2024-01-15T14:36:00.174645+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:06:57.295266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174327+0000","last_change":"2024-01-15T14:35:59.181518+0000","last_active":"2024-01-15T14:36:00.174327+0000","last_peered":"2024-01-15T14:36:00.174327+0000","last_clean":"2024-01-15T14:36:00.174327+0000","last_became_active":"2024-01-15T14:35:59.181265+0000","last_became_peered":"2024-01-15T14:35:59.181265+0000","last_unstale":"2024-01-15T14:36:00.174327+0000","last_undegraded":"2024-01-15T14:36:00.174327+0000","last_fullsized":"2024-01-15T14:36:00.174327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:06.345458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446593+0000","last_change":"2024-01-15T14:35:59.189569+0000","last_active":"2024-01-15T14:36:01.446593+0000","last_peered":"2024-01-15T14:36:01.446593+0000","last_clean":"2024-01-15T14:36:01.446593+0000","last_became_active":"2024-01-15T14:35:59.189328+0000","last_became_peered":"2024-01-15T14:35:59.189328+0000","last_unstale":"2024-01-15T14:36:01.446593+0000","last_undegraded":"2024-01-15T14:36:01.446593+0000","last_fullsized":"2024-01-15T14:36:01.446593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:27.127703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445298+0000","last_change":"2024-01-15T14:35:59.195328+0000","last_active":"2024-01-15T14:36:01.445298+0000","last_peered":"2024-01-15T14:36:01.445298+0000","last_clean":"2024-01-15T14:36:01.445298+0000","last_became_active":"2024-01-15T14:35:59.195042+0000","last_became_peered":"2024-01-15T14:35:59.195042+0000","last_unstale":"2024-01-15T14:36:01.445298+0000","last_undegraded":"2024-01-15T14:36:01.445298+0000","last_fullsized":"2024-01-15T14:36:01.445298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:47.775492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445309+0000","last_change":"2024-01-15T14:35:59.196400+0000","last_active":"2024-01-15T14:36:01.445309+0000","last_peered":"2024-01-15T14:36:01.445309+0000","last_clean":"2024-01-15T14:36:01.445309+0000","last_became_active":"2024-01-15T14:35:59.196079+0000","last_became_peered":"2024-01-15T14:35:59.196079+0000","last_unstale":"2024-01-15T14:36:01.445309+0000","last_undegraded":"2024-01-15T14:36:01.445309+0000","last_fullsized":"2024-01-15T14:36:01.445309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:19.359918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445633+0000","last_change":"2024-01-15T14:35:59.190292+0000","last_active":"2024-01-15T14:36:01.445633+0000","last_peered":"2024-01-15T14:36:01.445633+0000","last_clean":"2024-01-15T14:36:01.445633+0000","last_became_active":"2024-01-15T14:35:59.190035+0000","last_became_peered":"2024-01-15T14:35:59.190035+0000","last_unstale":"2024-01-15T14:36:01.445633+0000","last_undegraded":"2024-01-15T14:36:01.445633+0000","last_fullsized":"2024-01-15T14:36:01.445633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:49.856872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.344966+0000","last_change":"2024-01-15T14:35:59.188960+0000","last_active":"2024-01-15T14:36:01.344966+0000","last_peered":"2024-01-15T14:36:01.344966+0000","last_clean":"2024-01-15T14:36:01.344966+0000","last_became_active":"2024-01-15T14:35:59.188673+0000","last_became_peered":"2024-01-15T14:35:59.188673+0000","last_unstale":"2024-01-15T14:36:01.344966+0000","last_undegraded":"2024-01-15T14:36:01.344966+0000","last_fullsized":"2024-01-15T14:36:01.344966+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446674+0000","last_change":"2024-01-15T14:35:59.187507+0000","last_active":"2024-01-15T14:36:01.446674+0000","last_peered":"2024-01-15T14:36:01.446674+0000","last_clean":"2024-01-15T14:36:01.446674+0000","last_became_active":"2024-01-15T14:35:59.187262+0000","last_became_peered":"2024-01-15T14:35:59.187262+0000","last_unstale":"2024-01-15T14:36:01.446674+0000","last_undegraded":"2024-01-15T14:36:01.446674+0000","last_fullsized":"2024-01-15T14:36:01.446674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:30:42.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174669+0000","last_change":"2024-01-15T14:35:59.187360+0000","last_active":"2024-01-15T14:36:00.174669+0000","last_peered":"2024-01-15T14:36:00.174669+0000","last_clean":"2024-01-15T14:36:00.174669+0000","last_became_active":"2024-01-15T14:35:59.187108+0000","last_became_peered":"2024-01-15T14:35:59.187108+0000","last_unstale":"2024-01-15T14:36:00.174669+0000","last_undegraded":"2024-01-15T14:36:00.174669+0000","last_fullsized":"2024-01-15T14:36:00.174669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:21:10.420954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446670+0000","last_change":"2024-01-15T14:35:59.196017+0000","last_active":"2024-01-15T14:36:01.446670+0000","last_peered":"2024-01-15T14:36:01.446670+0000","last_clean":"2024-01-15T14:36:01.446670+0000","last_became_active":"2024-01-15T14:35:59.195845+0000","last_became_peered":"2024-01-15T14:35:59.195845+0000","last_unstale":"2024-01-15T14:36:01.446670+0000","last_undegraded":"2024-01-15T14:36:01.446670+0000","last_fullsized":"2024-01-15T14:36:01.446670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345260+0000","last_change":"2024-01-15T14:35:59.182114+0000","last_active":"2024-01-15T14:36:01.345260+0000","last_peered":"2024-01-15T14:36:01.345260+0000","last_clean":"2024-01-15T14:36:01.345260+0000","last_became_active":"2024-01-15T14:35:59.181778+0000","last_became_peered":"2024-01-15T14:35:59.181778+0000","last_unstale":"2024-01-15T14:36:01.345260+0000","last_undegraded":"2024-01-15T14:36:01.345260+0000","last_fullsized":"2024-01-15T14:36:01.345260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:06.345458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445461+0000","last_change":"2024-01-15T14:35:59.196390+0000","last_active":"2024-01-15T14:36:01.445461+0000","last_peered":"2024-01-15T14:36:01.445461+0000","last_clean":"2024-01-15T14:36:01.445461+0000","last_became_active":"2024-01-15T14:35:59.196098+0000","last_became_peered":"2024-01-15T14:35:59.196098+0000","last_unstale":"2024-01-15T14:36:01.445461+0000","last_undegraded":"2024-01-15T14:36:01.445461+0000","last_fullsized":"2024-01-15T14:36:01.445461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:30.135562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177783+0000","last_change":"2024-01-15T14:35:59.191955+0000","last_active":"2024-01-15T14:36:00.177783+0000","last_peered":"2024-01-15T14:36:00.177783+0000","last_clean":"2024-01-15T14:36:00.177783+0000","last_became_active":"2024-01-15T14:35:59.191519+0000","last_became_peered":"2024-01-15T14:35:59.191519+0000","last_unstale":"2024-01-15T14:36:00.177783+0000","last_undegraded":"2024-01-15T14:36:00.177783+0000","last_fullsized":"2024-01-15T14:36:00.177783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:44.088023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178297+0000","last_change":"2024-01-15T14:35:59.190402+0000","last_active":"2024-01-15T14:36:00.178297+0000","last_peered":"2024-01-15T14:36:00.178297+0000","last_clean":"2024-01-15T14:36:00.178297+0000","last_became_active":"2024-01-15T14:35:59.190116+0000","last_became_peered":"2024-01-15T14:35:59.190116+0000","last_unstale":"2024-01-15T14:36:00.178297+0000","last_undegraded":"2024-01-15T14:36:00.178297+0000","last_fullsized":"2024-01-15T14:36:00.178297+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:04.599483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178117+0000","last_change":"2024-01-15T14:35:59.195020+0000","last_active":"2024-01-15T14:36:00.178117+0000","last_peered":"2024-01-15T14:36:00.178117+0000","last_clean":"2024-01-15T14:36:00.178117+0000","last_became_active":"2024-01-15T14:35:59.194742+0000","last_became_peered":"2024-01-15T14:35:59.194742+0000","last_unstale":"2024-01-15T14:36:00.178117+0000","last_undegraded":"2024-01-15T14:36:00.178117+0000","last_fullsized":"2024-01-15T14:36:00.178117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:06.345458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446418+0000","last_change":"2024-01-15T14:35:59.187297+0000","last_active":"2024-01-15T14:36:01.446418+0000","last_peered":"2024-01-15T14:36:01.446418+0000","last_clean":"2024-01-15T14:36:01.446418+0000","last_became_active":"2024-01-15T14:35:59.186991+0000","last_became_peered":"2024-01-15T14:35:59.186991+0000","last_unstale":"2024-01-15T14:36:01.446418+0000","last_undegraded":"2024-01-15T14:36:01.446418+0000","last_fullsized":"2024-01-15T14:36:01.446418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:08.359295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178351+0000","last_change":"2024-01-15T14:35:58.715678+0000","last_active":"2024-01-15T14:36:00.178351+0000","last_peered":"2024-01-15T14:36:00.178351+0000","last_clean":"2024-01-15T14:36:00.178351+0000","last_became_active":"2024-01-15T14:35:58.715392+0000","last_became_peered":"2024-01-15T14:35:58.715392+0000","last_unstale":"2024-01-15T14:36:00.178351+0000","last_undegraded":"2024-01-15T14:36:00.178351+0000","last_fullsized":"2024-01-15T14:36:00.178351+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:45:53.496590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178265+0000","last_change":"2024-01-15T14:35:59.196451+0000","last_active":"2024-01-15T14:36:00.178265+0000","last_peered":"2024-01-15T14:36:00.178265+0000","last_clean":"2024-01-15T14:36:00.178265+0000","last_became_active":"2024-01-15T14:35:59.196237+0000","last_became_peered":"2024-01-15T14:35:59.196237+0000","last_unstale":"2024-01-15T14:36:00.178265+0000","last_undegraded":"2024-01-15T14:36:00.178265+0000","last_fullsized":"2024-01-15T14:36:00.178265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:01.066110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446468+0000","last_change":"2024-01-15T14:35:58.718612+0000","last_active":"2024-01-15T14:36:01.446468+0000","last_peered":"2024-01-15T14:36:01.446468+0000","last_clean":"2024-01-15T14:36:01.446468+0000","last_became_active":"2024-01-15T14:35:58.718350+0000","last_became_peered":"2024-01-15T14:35:58.718350+0000","last_unstale":"2024-01-15T14:36:01.446468+0000","last_undegraded":"2024-01-15T14:36:01.446468+0000","last_fullsized":"2024-01-15T14:36:01.446468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:29:50.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445464+0000","last_change":"2024-01-15T14:35:59.194990+0000","last_active":"2024-01-15T14:36:01.445464+0000","last_peered":"2024-01-15T14:36:01.445464+0000","last_clean":"2024-01-15T14:36:01.445464+0000","last_became_active":"2024-01-15T14:35:59.194701+0000","last_became_peered":"2024-01-15T14:35:59.194701+0000","last_unstale":"2024-01-15T14:36:01.445464+0000","last_undegraded":"2024-01-15T14:36:01.445464+0000","last_fullsized":"2024-01-15T14:36:01.445464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:18:13.350946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174596+0000","last_change":"2024-01-15T14:35:58.719422+0000","last_active":"2024-01-15T14:36:00.174596+0000","last_peered":"2024-01-15T14:36:00.174596+0000","last_clean":"2024-01-15T14:36:00.174596+0000","last_became_active":"2024-01-15T14:35:58.718448+0000","last_became_peered":"2024-01-15T14:35:58.718448+0000","last_unstale":"2024-01-15T14:36:00.174596+0000","last_undegraded":"2024-01-15T14:36:00.174596+0000","last_fullsized":"2024-01-15T14:36:00.174596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:46.766398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446870+0000","last_change":"2024-01-15T14:35:59.196067+0000","last_active":"2024-01-15T14:36:01.446870+0000","last_peered":"2024-01-15T14:36:01.446870+0000","last_clean":"2024-01-15T14:36:01.446870+0000","last_became_active":"2024-01-15T14:35:59.195874+0000","last_became_peered":"2024-01-15T14:35:59.195874+0000","last_unstale":"2024-01-15T14:36:01.446870+0000","last_undegraded":"2024-01-15T14:36:01.446870+0000","last_fullsized":"2024-01-15T14:36:01.446870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446318+0000","last_change":"2024-01-15T14:35:58.716192+0000","last_active":"2024-01-15T14:36:01.446318+0000","last_peered":"2024-01-15T14:36:01.446318+0000","last_clean":"2024-01-15T14:36:01.446318+0000","last_became_active":"2024-01-15T14:35:58.715905+0000","last_became_peered":"2024-01-15T14:35:58.715905+0000","last_unstale":"2024-01-15T14:36:01.446318+0000","last_undegraded":"2024-01-15T14:36:01.446318+0000","last_fullsized":"2024-01-15T14:36:01.446318+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:14.739333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177935+0000","last_change":"2024-01-15T14:35:59.197382+0000","last_active":"2024-01-15T14:36:00.177935+0000","last_peered":"2024-01-15T14:36:00.177935+0000","last_clean":"2024-01-15T14:36:00.177935+0000","last_became_active":"2024-01-15T14:35:59.197103+0000","last_became_peered":"2024-01-15T14:35:59.197103+0000","last_unstale":"2024-01-15T14:36:00.177935+0000","last_undegraded":"2024-01-15T14:36:00.177935+0000","last_fullsized":"2024-01-15T14:36:00.177935+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:30:42.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445397+0000","last_change":"2024-01-15T14:35:58.718597+0000","last_active":"2024-01-15T14:36:01.445397+0000","last_peered":"2024-01-15T14:36:01.445397+0000","last_clean":"2024-01-15T14:36:01.445397+0000","last_became_active":"2024-01-15T14:35:58.718339+0000","last_became_peered":"2024-01-15T14:35:58.718339+0000","last_unstale":"2024-01-15T14:36:01.445397+0000","last_undegraded":"2024-01-15T14:36:01.445397+0000","last_fullsized":"2024-01-15T14:36:01.445397+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:48.847777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345340+0000","last_change":"2024-01-15T14:35:59.188867+0000","last_active":"2024-01-15T14:36:01.345340+0000","last_peered":"2024-01-15T14:36:01.345340+0000","last_clean":"2024-01-15T14:36:01.345340+0000","last_became_active":"2024-01-15T14:35:59.186904+0000","last_became_peered":"2024-01-15T14:35:59.186904+0000","last_unstale":"2024-01-15T14:36:01.345340+0000","last_undegraded":"2024-01-15T14:36:01.345340+0000","last_fullsized":"2024-01-15T14:36:01.345340+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:04.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445238+0000","last_change":"2024-01-15T14:35:58.720981+0000","last_active":"2024-01-15T14:36:01.445238+0000","last_peered":"2024-01-15T14:36:01.445238+0000","last_clean":"2024-01-15T14:36:01.445238+0000","last_became_active":"2024-01-15T14:35:58.720645+0000","last_became_peered":"2024-01-15T14:35:58.720645+0000","last_unstale":"2024-01-15T14:36:01.445238+0000","last_undegraded":"2024-01-15T14:36:01.445238+0000","last_fullsized":"2024-01-15T14:36:01.445238+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:23:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445256+0000","last_change":"2024-01-15T14:35:59.199041+0000","last_active":"2024-01-15T14:36:01.445256+0000","last_peered":"2024-01-15T14:36:01.445256+0000","last_clean":"2024-01-15T14:36:01.445256+0000","last_became_active":"2024-01-15T14:35:59.198728+0000","last_became_peered":"2024-01-15T14:35:59.198728+0000","last_unstale":"2024-01-15T14:36:01.445256+0000","last_undegraded":"2024-01-15T14:36:01.445256+0000","last_fullsized":"2024-01-15T14:36:01.445256+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:02:35.153060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174081+0000","last_change":"2024-01-15T14:35:58.724180+0000","last_active":"2024-01-15T14:36:00.174081+0000","last_peered":"2024-01-15T14:36:00.174081+0000","last_clean":"2024-01-15T14:36:00.174081+0000","last_became_active":"2024-01-15T14:35:58.723873+0000","last_became_peered":"2024-01-15T14:35:58.723873+0000","last_unstale":"2024-01-15T14:36:00.174081+0000","last_undegraded":"2024-01-15T14:36:00.174081+0000","last_fullsized":"2024-01-15T14:36:00.174081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:00.057015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445805+0000","last_change":"2024-01-15T14:35:59.184327+0000","last_active":"2024-01-15T14:36:01.445805+0000","last_peered":"2024-01-15T14:36:01.445805+0000","last_clean":"2024-01-15T14:36:01.445805+0000","last_became_active":"2024-01-15T14:35:59.184112+0000","last_became_peered":"2024-01-15T14:35:59.184112+0000","last_unstale":"2024-01-15T14:36:01.445805+0000","last_undegraded":"2024-01-15T14:36:01.445805+0000","last_fullsized":"2024-01-15T14:36:01.445805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:47.775492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445144+0000","last_change":"2024-01-15T14:35:58.717974+0000","last_active":"2024-01-15T14:36:01.445144+0000","last_peered":"2024-01-15T14:36:01.445144+0000","last_clean":"2024-01-15T14:36:01.445144+0000","last_became_active":"2024-01-15T14:35:58.717679+0000","last_became_peered":"2024-01-15T14:35:58.717679+0000","last_unstale":"2024-01-15T14:36:01.445144+0000","last_undegraded":"2024-01-15T14:36:01.445144+0000","last_fullsized":"2024-01-15T14:36:01.445144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:40.876364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345413+0000","last_change":"2024-01-15T14:35:59.188560+0000","last_active":"2024-01-15T14:36:01.345413+0000","last_peered":"2024-01-15T14:36:01.345413+0000","last_clean":"2024-01-15T14:36:01.345413+0000","last_became_active":"2024-01-15T14:35:59.187878+0000","last_became_peered":"2024-01-15T14:35:59.187878+0000","last_unstale":"2024-01-15T14:36:01.345413+0000","last_undegraded":"2024-01-15T14:36:01.345413+0000","last_fullsized":"2024-01-15T14:36:01.345413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:40.243794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174372+0000","last_change":"2024-01-15T14:35:58.722504+0000","last_active":"2024-01-15T14:36:00.174372+0000","last_peered":"2024-01-15T14:36:00.174372+0000","last_clean":"2024-01-15T14:36:00.174372+0000","last_became_active":"2024-01-15T14:35:58.722227+0000","last_became_peered":"2024-01-15T14:35:58.722227+0000","last_unstale":"2024-01-15T14:36:00.174372+0000","last_undegraded":"2024-01-15T14:36:00.174372+0000","last_fullsized":"2024-01-15T14:36:00.174372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:31:41.196900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174525+0000","last_change":"2024-01-15T14:35:59.180752+0000","last_active":"2024-01-15T14:36:00.174525+0000","last_peered":"2024-01-15T14:36:00.174525+0000","last_clean":"2024-01-15T14:36:00.174525+0000","last_became_active":"2024-01-15T14:35:59.180504+0000","last_became_peered":"2024-01-15T14:35:59.180504+0000","last_unstale":"2024-01-15T14:36:00.174525+0000","last_undegraded":"2024-01-15T14:36:00.174525+0000","last_fullsized":"2024-01-15T14:36:00.174525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:21:21.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446502+0000","last_change":"2024-01-15T14:35:58.721258+0000","last_active":"2024-01-15T14:36:01.446502+0000","last_peered":"2024-01-15T14:36:01.446502+0000","last_clean":"2024-01-15T14:36:01.446502+0000","last_became_active":"2024-01-15T14:35:58.720974+0000","last_became_peered":"2024-01-15T14:35:58.720974+0000","last_unstale":"2024-01-15T14:36:01.446502+0000","last_undegraded":"2024-01-15T14:36:01.446502+0000","last_fullsized":"2024-01-15T14:36:01.446502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:23:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174760+0000","last_change":"2024-01-15T14:35:59.188093+0000","last_active":"2024-01-15T14:36:00.174760+0000","last_peered":"2024-01-15T14:36:00.174760+0000","last_clean":"2024-01-15T14:36:00.174760+0000","last_became_active":"2024-01-15T14:35:59.187810+0000","last_became_peered":"2024-01-15T14:35:59.187810+0000","last_unstale":"2024-01-15T14:36:00.174760+0000","last_undegraded":"2024-01-15T14:36:00.174760+0000","last_fullsized":"2024-01-15T14:36:00.174760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:59.382998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345317+0000","last_change":"2024-01-15T14:35:58.720133+0000","last_active":"2024-01-15T14:36:01.345317+0000","last_peered":"2024-01-15T14:36:01.345317+0000","last_clean":"2024-01-15T14:36:01.345317+0000","last_became_active":"2024-01-15T14:35:58.719855+0000","last_became_peered":"2024-01-15T14:35:58.719855+0000","last_unstale":"2024-01-15T14:36:01.345317+0000","last_undegraded":"2024-01-15T14:36:01.345317+0000","last_fullsized":"2024-01-15T14:36:01.345317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:29.126468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345358+0000","last_change":"2024-01-15T14:35:59.187707+0000","last_active":"2024-01-15T14:36:01.345358+0000","last_peered":"2024-01-15T14:36:01.345358+0000","last_clean":"2024-01-15T14:36:01.345358+0000","last_became_active":"2024-01-15T14:35:59.187081+0000","last_became_peered":"2024-01-15T14:35:59.187081+0000","last_unstale":"2024-01-15T14:36:01.345358+0000","last_undegraded":"2024-01-15T14:36:01.345358+0000","last_fullsized":"2024-01-15T14:36:01.345358+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:37.731830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174825+0000","last_change":"2024-01-15T14:35:58.719318+0000","last_active":"2024-01-15T14:36:00.174825+0000","last_peered":"2024-01-15T14:36:00.174825+0000","last_clean":"2024-01-15T14:36:00.174825+0000","last_became_active":"2024-01-15T14:35:58.719038+0000","last_became_peered":"2024-01-15T14:35:58.719038+0000","last_unstale":"2024-01-15T14:36:00.174825+0000","last_undegraded":"2024-01-15T14:36:00.174825+0000","last_fullsized":"2024-01-15T14:36:00.174825+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:43.351804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445921+0000","last_change":"2024-01-15T14:35:59.190039+0000","last_active":"2024-01-15T14:36:01.445921+0000","last_peered":"2024-01-15T14:36:01.445921+0000","last_clean":"2024-01-15T14:36:01.445921+0000","last_became_active":"2024-01-15T14:35:59.189775+0000","last_became_peered":"2024-01-15T14:35:59.189775+0000","last_unstale":"2024-01-15T14:36:01.445921+0000","last_undegraded":"2024-01-15T14:36:01.445921+0000","last_fullsized":"2024-01-15T14:36:01.445921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:19.359918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445513+0000","last_change":"2024-01-15T14:35:58.708011+0000","last_active":"2024-01-15T14:36:01.445513+0000","last_peered":"2024-01-15T14:36:01.445513+0000","last_clean":"2024-01-15T14:36:01.445513+0000","last_became_active":"2024-01-15T14:35:58.707767+0000","last_became_peered":"2024-01-15T14:35:58.707767+0000","last_unstale":"2024-01-15T14:36:01.445513+0000","last_undegraded":"2024-01-15T14:36:01.445513+0000","last_fullsized":"2024-01-15T14:36:01.445513+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:07.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174375+0000","last_change":"2024-01-15T14:35:59.182071+0000","last_active":"2024-01-15T14:36:00.174375+0000","last_peered":"2024-01-15T14:36:00.174375+0000","last_clean":"2024-01-15T14:36:00.174375+0000","last_became_active":"2024-01-15T14:35:59.181815+0000","last_became_peered":"2024-01-15T14:35:59.181815+0000","last_unstale":"2024-01-15T14:36:00.174375+0000","last_undegraded":"2024-01-15T14:36:00.174375+0000","last_fullsized":"2024-01-15T14:36:00.174375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:04.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446019+0000","last_change":"2024-01-15T14:35:58.720466+0000","last_active":"2024-01-15T14:36:01.446019+0000","last_peered":"2024-01-15T14:36:01.446019+0000","last_clean":"2024-01-15T14:36:01.446019+0000","last_became_active":"2024-01-15T14:35:58.720193+0000","last_became_peered":"2024-01-15T14:35:58.720193+0000","last_unstale":"2024-01-15T14:36:01.446019+0000","last_undegraded":"2024-01-15T14:36:01.446019+0000","last_fullsized":"2024-01-15T14:36:01.446019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:21.624021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177972+0000","last_change":"2024-01-15T14:35:59.197290+0000","last_active":"2024-01-15T14:36:00.177972+0000","last_peered":"2024-01-15T14:36:00.177972+0000","last_clean":"2024-01-15T14:36:00.177972+0000","last_became_active":"2024-01-15T14:35:59.197015+0000","last_became_peered":"2024-01-15T14:35:59.197015+0000","last_unstale":"2024-01-15T14:36:00.177972+0000","last_undegraded":"2024-01-15T14:36:00.177972+0000","last_fullsized":"2024-01-15T14:36:00.177972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174155+0000","last_change":"2024-01-15T14:35:58.722915+0000","last_active":"2024-01-15T14:36:00.174155+0000","last_peered":"2024-01-15T14:36:00.174155+0000","last_clean":"2024-01-15T14:36:00.174155+0000","last_became_active":"2024-01-15T14:35:58.722300+0000","last_became_peered":"2024-01-15T14:35:58.722300+0000","last_unstale":"2024-01-15T14:36:00.174155+0000","last_undegraded":"2024-01-15T14:36:00.174155+0000","last_fullsized":"2024-01-15T14:36:00.174155+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:26.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178145+0000","last_change":"2024-01-15T14:35:59.194805+0000","last_active":"2024-01-15T14:36:00.178145+0000","last_peered":"2024-01-15T14:36:00.178145+0000","last_clean":"2024-01-15T14:36:00.178145+0000","last_became_active":"2024-01-15T14:35:59.194545+0000","last_became_peered":"2024-01-15T14:35:59.194545+0000","last_unstale":"2024-01-15T14:36:00.178145+0000","last_undegraded":"2024-01-15T14:36:00.178145+0000","last_fullsized":"2024-01-15T14:36:00.178145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:27.154968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174032+0000","last_change":"2024-01-15T14:35:58.718692+0000","last_active":"2024-01-15T14:36:00.174032+0000","last_peered":"2024-01-15T14:36:00.174032+0000","last_clean":"2024-01-15T14:36:00.174032+0000","last_became_active":"2024-01-15T14:35:58.718433+0000","last_became_peered":"2024-01-15T14:35:58.718433+0000","last_unstale":"2024-01-15T14:36:00.174032+0000","last_undegraded":"2024-01-15T14:36:00.174032+0000","last_fullsized":"2024-01-15T14:36:00.174032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:21.191583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445652+0000","last_change":"2024-01-15T14:35:59.195055+0000","last_active":"2024-01-15T14:36:01.445652+0000","last_peered":"2024-01-15T14:36:01.445652+0000","last_clean":"2024-01-15T14:36:01.445652+0000","last_became_active":"2024-01-15T14:35:59.194468+0000","last_became_peered":"2024-01-15T14:35:59.194468+0000","last_unstale":"2024-01-15T14:36:01.445652+0000","last_undegraded":"2024-01-15T14:36:01.445652+0000","last_fullsized":"2024-01-15T14:36:01.445652+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446779+0000","last_change":"2024-01-15T14:35:58.722370+0000","last_active":"2024-01-15T14:36:01.446779+0000","last_peered":"2024-01-15T14:36:01.446779+0000","last_clean":"2024-01-15T14:36:01.446779+0000","last_became_active":"2024-01-15T14:35:58.722085+0000","last_became_peered":"2024-01-15T14:35:58.722085+0000","last_unstale":"2024-01-15T14:36:01.446779+0000","last_undegraded":"2024-01-15T14:36:01.446779+0000","last_fullsized":"2024-01-15T14:36:01.446779+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:31:41.196900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345435+0000","last_change":"2024-01-15T14:35:59.195889+0000","last_active":"2024-01-15T14:36:01.345435+0000","last_peered":"2024-01-15T14:36:01.345435+0000","last_clean":"2024-01-15T14:36:01.345435+0000","last_became_active":"2024-01-15T14:35:59.195574+0000","last_became_peered":"2024-01-15T14:35:59.195574+0000","last_unstale":"2024-01-15T14:36:01.345435+0000","last_undegraded":"2024-01-15T14:36:01.345435+0000","last_fullsized":"2024-01-15T14:36:01.345435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:21:21.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445615+0000","last_change":"2024-01-15T14:35:58.708707+0000","last_active":"2024-01-15T14:36:01.445615+0000","last_peered":"2024-01-15T14:36:01.445615+0000","last_clean":"2024-01-15T14:36:01.445615+0000","last_became_active":"2024-01-15T14:35:58.708419+0000","last_became_peered":"2024-01-15T14:35:58.708419+0000","last_unstale":"2024-01-15T14:36:01.445615+0000","last_undegraded":"2024-01-15T14:36:01.445615+0000","last_fullsized":"2024-01-15T14:36:01.445615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:26.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446511+0000","last_change":"2024-01-15T14:35:59.196118+0000","last_active":"2024-01-15T14:36:01.446511+0000","last_peered":"2024-01-15T14:36:01.446511+0000","last_clean":"2024-01-15T14:36:01.446511+0000","last_became_active":"2024-01-15T14:35:59.195858+0000","last_became_peered":"2024-01-15T14:35:59.195858+0000","last_unstale":"2024-01-15T14:36:01.446511+0000","last_undegraded":"2024-01-15T14:36:01.446511+0000","last_fullsized":"2024-01-15T14:36:01.446511+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:44.360899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445596+0000","last_change":"2024-01-15T14:35:58.724003+0000","last_active":"2024-01-15T14:36:01.445596+0000","last_peered":"2024-01-15T14:36:01.445596+0000","last_clean":"2024-01-15T14:36:01.445596+0000","last_became_active":"2024-01-15T14:35:58.723747+0000","last_became_peered":"2024-01-15T14:35:58.723747+0000","last_unstale":"2024-01-15T14:36:01.445596+0000","last_undegraded":"2024-01-15T14:36:01.445596+0000","last_fullsized":"2024-01-15T14:36:01.445596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:03.590388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446044+0000","last_change":"2024-01-15T14:35:59.191930+0000","last_active":"2024-01-15T14:36:01.446044+0000","last_peered":"2024-01-15T14:36:01.446044+0000","last_clean":"2024-01-15T14:36:01.446044+0000","last_became_active":"2024-01-15T14:35:59.191779+0000","last_became_peered":"2024-01-15T14:35:59.191779+0000","last_unstale":"2024-01-15T14:36:01.446044+0000","last_undegraded":"2024-01-15T14:36:01.446044+0000","last_fullsized":"2024-01-15T14:36:01.446044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:29:51.441617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445531+0000","last_change":"2024-01-15T14:35:58.716285+0000","last_active":"2024-01-15T14:36:01.445531+0000","last_peered":"2024-01-15T14:36:01.445531+0000","last_clean":"2024-01-15T14:36:01.445531+0000","last_became_active":"2024-01-15T14:35:58.716016+0000","last_became_peered":"2024-01-15T14:35:58.716016+0000","last_unstale":"2024-01-15T14:36:01.445531+0000","last_undegraded":"2024-01-15T14:36:01.445531+0000","last_fullsized":"2024-01-15T14:36:01.445531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:21.624021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446762+0000","last_change":"2024-01-15T14:35:59.197943+0000","last_active":"2024-01-15T14:36:01.446762+0000","last_peered":"2024-01-15T14:36:01.446762+0000","last_clean":"2024-01-15T14:36:01.446762+0000","last_became_active":"2024-01-15T14:35:59.197679+0000","last_became_peered":"2024-01-15T14:35:59.197679+0000","last_unstale":"2024-01-15T14:36:01.446762+0000","last_undegraded":"2024-01-15T14:36:01.446762+0000","last_fullsized":"2024-01-15T14:36:01.446762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:27.154968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446107+0000","last_change":"2024-01-15T14:35:58.719317+0000","last_active":"2024-01-15T14:36:01.446107+0000","last_peered":"2024-01-15T14:36:01.446107+0000","last_clean":"2024-01-15T14:36:01.446107+0000","last_became_active":"2024-01-15T14:35:58.719069+0000","last_became_peered":"2024-01-15T14:35:58.719069+0000","last_unstale":"2024-01-15T14:36:01.446107+0000","last_undegraded":"2024-01-15T14:36:01.446107+0000","last_fullsized":"2024-01-15T14:36:01.446107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:00.057015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446037+0000","last_change":"2024-01-15T14:35:59.190777+0000","last_active":"2024-01-15T14:36:01.446037+0000","last_peered":"2024-01-15T14:36:01.446037+0000","last_clean":"2024-01-15T14:36:01.446037+0000","last_became_active":"2024-01-15T14:35:59.190526+0000","last_became_peered":"2024-01-15T14:35:59.190526+0000","last_unstale":"2024-01-15T14:36:01.446037+0000","last_undegraded":"2024-01-15T14:36:01.446037+0000","last_fullsized":"2024-01-15T14:36:01.446037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:45:54.505684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178709+0000","last_change":"2024-01-15T14:35:58.715685+0000","last_active":"2024-01-15T14:36:00.178709+0000","last_peered":"2024-01-15T14:36:00.178709+0000","last_clean":"2024-01-15T14:36:00.178709+0000","last_became_active":"2024-01-15T14:35:58.715406+0000","last_became_peered":"2024-01-15T14:35:58.715406+0000","last_unstale":"2024-01-15T14:36:00.178709+0000","last_undegraded":"2024-01-15T14:36:00.178709+0000","last_fullsized":"2024-01-15T14:36:00.178709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T16:51:26.118609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445469+0000","last_change":"2024-01-15T14:35:59.190741+0000","last_active":"2024-01-15T14:36:01.445469+0000","last_peered":"2024-01-15T14:36:01.445469+0000","last_clean":"2024-01-15T14:36:01.445469+0000","last_became_active":"2024-01-15T14:35:59.190474+0000","last_became_peered":"2024-01-15T14:35:59.190474+0000","last_unstale":"2024-01-15T14:36:01.445469+0000","last_undegraded":"2024-01-15T14:36:01.445469+0000","last_fullsized":"2024-01-15T14:36:01.445469+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:35:47.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445241+0000","last_change":"2024-01-15T14:35:58.715057+0000","last_active":"2024-01-15T14:36:01.445241+0000","last_peered":"2024-01-15T14:36:01.445241+0000","last_clean":"2024-01-15T14:36:01.445241+0000","last_became_active":"2024-01-15T14:35:58.714781+0000","last_became_peered":"2024-01-15T14:35:58.714781+0000","last_unstale":"2024-01-15T14:36:01.445241+0000","last_undegraded":"2024-01-15T14:36:01.445241+0000","last_fullsized":"2024-01-15T14:36:01.445241+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:21.191583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178728+0000","last_change":"2024-01-15T14:35:59.192344+0000","last_active":"2024-01-15T14:36:00.178728+0000","last_peered":"2024-01-15T14:36:00.178728+0000","last_clean":"2024-01-15T14:36:00.178728+0000","last_became_active":"2024-01-15T14:35:59.191891+0000","last_became_peered":"2024-01-15T14:35:59.191891+0000","last_unstale":"2024-01-15T14:36:00.178728+0000","last_undegraded":"2024-01-15T14:36:00.178728+0000","last_fullsized":"2024-01-15T14:36:00.178728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:21:21.695785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.344911+0000","last_change":"2024-01-15T14:35:58.716350+0000","last_active":"2024-01-15T14:36:01.344911+0000","last_peered":"2024-01-15T14:36:01.344911+0000","last_clean":"2024-01-15T14:36:01.344911+0000","last_became_active":"2024-01-15T14:35:58.715634+0000","last_became_peered":"2024-01-15T14:35:58.715634+0000","last_unstale":"2024-01-15T14:36:01.344911+0000","last_undegraded":"2024-01-15T14:36:01.344911+0000","last_fullsized":"2024-01-15T14:36:01.344911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:12:18.350823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446640+0000","last_change":"2024-01-15T14:35:59.189401+0000","last_active":"2024-01-15T14:36:01.446640+0000","last_peered":"2024-01-15T14:36:01.446640+0000","last_clean":"2024-01-15T14:36:01.446640+0000","last_became_active":"2024-01-15T14:35:59.188947+0000","last_became_peered":"2024-01-15T14:35:59.188947+0000","last_unstale":"2024-01-15T14:36:01.446640+0000","last_undegraded":"2024-01-15T14:36:01.446640+0000","last_fullsized":"2024-01-15T14:36:01.446640+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:22.200678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446626+0000","last_change":"2024-01-15T14:35:58.720184+0000","last_active":"2024-01-15T14:36:01.446626+0000","last_peered":"2024-01-15T14:36:01.446626+0000","last_clean":"2024-01-15T14:36:01.446626+0000","last_became_active":"2024-01-15T14:35:58.719875+0000","last_became_peered":"2024-01-15T14:35:58.719875+0000","last_unstale":"2024-01-15T14:36:01.446626+0000","last_undegraded":"2024-01-15T14:36:01.446626+0000","last_fullsized":"2024-01-15T14:36:01.446626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:07.350201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.345544+0000","last_change":"2024-01-15T14:35:59.188314+0000","last_active":"2024-01-15T14:36:01.345544+0000","last_peered":"2024-01-15T14:36:01.345544+0000","last_clean":"2024-01-15T14:36:01.345544+0000","last_became_active":"2024-01-15T14:35:59.186735+0000","last_became_peered":"2024-01-15T14:35:59.186735+0000","last_unstale":"2024-01-15T14:36:01.345544+0000","last_undegraded":"2024-01-15T14:36:01.345544+0000","last_fullsized":"2024-01-15T14:36:01.345544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:00.462542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.738774+0000","last_change":"2024-01-15T14:35:58.719333+0000","last_active":"2024-01-15T14:36:01.738774+0000","last_peered":"2024-01-15T14:36:01.738774+0000","last_clean":"2024-01-15T14:36:01.738774+0000","last_became_active":"2024-01-15T14:35:58.719079+0000","last_became_peered":"2024-01-15T14:35:58.719079+0000","last_unstale":"2024-01-15T14:36:01.738774+0000","last_undegraded":"2024-01-15T14:36:01.738774+0000","last_fullsized":"2024-01-15T14:36:01.738774+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:21.191583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445755+0000","last_change":"2024-01-15T14:35:59.186650+0000","last_active":"2024-01-15T14:36:01.445755+0000","last_peered":"2024-01-15T14:36:01.445755+0000","last_clean":"2024-01-15T14:36:01.445755+0000","last_became_active":"2024-01-15T14:35:59.186396+0000","last_became_peered":"2024-01-15T14:35:59.186396+0000","last_unstale":"2024-01-15T14:36:01.445755+0000","last_undegraded":"2024-01-15T14:36:01.445755+0000","last_fullsized":"2024-01-15T14:36:01.445755+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:22.633116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446269+0000","last_change":"2024-01-15T14:35:58.724022+0000","last_active":"2024-01-15T14:36:01.446269+0000","last_peered":"2024-01-15T14:36:01.446269+0000","last_clean":"2024-01-15T14:36:01.446269+0000","last_became_active":"2024-01-15T14:35:58.723742+0000","last_became_peered":"2024-01-15T14:35:58.723742+0000","last_unstale":"2024-01-15T14:36:01.446269+0000","last_undegraded":"2024-01-15T14:36:01.446269+0000","last_fullsized":"2024-01-15T14:36:01.446269+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:40.876364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178619+0000","last_change":"2024-01-15T14:35:59.188111+0000","last_active":"2024-01-15T14:36:00.178619+0000","last_peered":"2024-01-15T14:36:00.178619+0000","last_clean":"2024-01-15T14:36:00.178619+0000","last_became_active":"2024-01-15T14:35:59.187805+0000","last_became_peered":"2024-01-15T14:35:59.187805+0000","last_unstale":"2024-01-15T14:36:00.178619+0000","last_undegraded":"2024-01-15T14:36:00.178619+0000","last_fullsized":"2024-01-15T14:36:00.178619+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:49.856872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446015+0000","last_change":"2024-01-15T14:35:58.715522+0000","last_active":"2024-01-15T14:36:01.446015+0000","last_peered":"2024-01-15T14:36:01.446015+0000","last_clean":"2024-01-15T14:36:01.446015+0000","last_became_active":"2024-01-15T14:35:58.715173+0000","last_became_peered":"2024-01-15T14:35:58.715173+0000","last_unstale":"2024-01-15T14:36:01.446015+0000","last_undegraded":"2024-01-15T14:36:01.446015+0000","last_fullsized":"2024-01-15T14:36:01.446015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:26.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446347+0000","last_change":"2024-01-15T14:35:59.186736+0000","last_active":"2024-01-15T14:36:01.446347+0000","last_peered":"2024-01-15T14:36:01.446347+0000","last_clean":"2024-01-15T14:36:01.446347+0000","last_became_active":"2024-01-15T14:35:59.186445+0000","last_became_peered":"2024-01-15T14:35:59.186445+0000","last_unstale":"2024-01-15T14:36:01.446347+0000","last_undegraded":"2024-01-15T14:36:01.446347+0000","last_fullsized":"2024-01-15T14:36:01.446347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:41.885459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.446717+0000","last_change":"2024-01-15T14:35:58.717510+0000","last_active":"2024-01-15T14:36:01.446717+0000","last_peered":"2024-01-15T14:36:01.446717+0000","last_clean":"2024-01-15T14:36:01.446717+0000","last_became_active":"2024-01-15T14:35:58.717199+0000","last_became_peered":"2024-01-15T14:35:58.717199+0000","last_unstale":"2024-01-15T14:36:01.446717+0000","last_undegraded":"2024-01-15T14:36:01.446717+0000","last_fullsized":"2024-01-15T14:36:01.446717+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:48.847777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174086+0000","last_change":"2024-01-15T14:35:59.181030+0000","last_active":"2024-01-15T14:36:00.174086+0000","last_peered":"2024-01-15T14:36:00.174086+0000","last_clean":"2024-01-15T14:36:00.174086+0000","last_became_active":"2024-01-15T14:35:59.180778+0000","last_became_peered":"2024-01-15T14:35:59.180778+0000","last_unstale":"2024-01-15T14:36:00.174086+0000","last_undegraded":"2024-01-15T14:36:00.174086+0000","last_fullsized":"2024-01-15T14:36:00.174086+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:53.875761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445317+0000","last_change":"2024-01-15T14:35:58.711993+0000","last_active":"2024-01-15T14:36:01.445317+0000","last_peered":"2024-01-15T14:36:01.445317+0000","last_clean":"2024-01-15T14:36:01.445317+0000","last_became_active":"2024-01-15T14:35:58.711721+0000","last_became_peered":"2024-01-15T14:35:58.711721+0000","last_unstale":"2024-01-15T14:36:01.445317+0000","last_undegraded":"2024-01-15T14:36:01.445317+0000","last_fullsized":"2024-01-15T14:36:01.445317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T15:08:00.057015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:01.445172+0000","last_change":"2024-01-15T14:35:59.195866+0000","last_active":"2024-01-15T14:36:01.445172+0000","last_peered":"2024-01-15T14:36:01.445172+0000","last_clean":"2024-01-15T14:36:01.445172+0000","last_became_active":"2024-01-15T14:35:59.195567+0000","last_became_peered":"2024-01-15T14:35:59.195567+0000","last_unstale":"2024-01-15T14:36:01.445172+0000","last_undegraded":"2024-01-15T14:36:01.445172+0000","last_fullsized":"2024-01-15T14:36:01.445172+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T17:31:42.205995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.174177+0000","last_change":"2024-01-15T14:35:58.722668+0000","last_active":"2024-01-15T14:36:00.174177+0000","last_peered":"2024-01-15T14:36:00.174177+0000","last_clean":"2024-01-15T14:36:00.174177+0000","last_became_active":"2024-01-15T14:35:58.722367+0000","last_became_peered":"2024-01-15T14:35:58.722367+0000","last_unstale":"2024-01-15T14:36:00.174177+0000","last_undegraded":"2024-01-15T14:36:00.174177+0000","last_fullsized":"2024-01-15T14:36:00.174177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:26:03.590388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.178221+0000","last_change":"2024-01-15T14:35:59.191746+0000","last_active":"2024-01-15T14:36:00.178221+0000","last_peered":"2024-01-15T14:36:00.178221+0000","last_clean":"2024-01-15T14:36:00.178221+0000","last_became_active":"2024-01-15T14:35:59.191440+0000","last_became_peered":"2024-01-15T14:35:59.191440+0000","last_unstale":"2024-01-15T14:36:00.178221+0000","last_undegraded":"2024-01-15T14:36:00.178221+0000","last_fullsized":"2024-01-15T14:36:00.178221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:58.696955+0000","last_clean_scrub_stamp":"2024-01-15T14:35:58.696955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:04.108882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T14:36:00.177843+0000","last_change":"2024-01-15T14:35:58.716125+0000","last_active":"2024-01-15T14:36:00.177843+0000","last_peered":"2024-01-15T14:36:00.177843+0000","last_clean":"2024-01-15T14:36:00.177843+0000","last_became_active":"2024-01-15T14:35:58.715847+0000","last_became_peered":"2024-01-15T14:35:58.715847+0000","last_unstale":"2024-01-15T14:36:00.177843+0000","last_undegraded":"2024-01-15T14:36:00.177843+0000","last_fullsized":"2024-01-15T14:36:00.177843+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:35:57.687860+0000","last_clean_scrub_stamp":"2024-01-15T14:35:57.687860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:14.739333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T14:35:59.181837+0000","last_change":"2024-01-15T14:34:54.738502+0000","last_active":"2024-01-15T14:35:59.181837+0000","last_peered":"2024-01-15T14:35:59.181837+0000","last_clean":"2024-01-15T14:35:59.181837+0000","last_became_active":"2024-01-15T14:34:54.737354+0000","last_became_peered":"2024-01-15T14:34:54.737354+0000","last_unstale":"2024-01-15T14:35:59.181837+0000","last_undegraded":"2024-01-15T14:35:59.181837+0000","last_fullsized":"2024-01-15T14:35:59.181837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T14:34:53.124419+0000","last_clean_scrub_stamp":"2024-01-15T14:34:53.124419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:09:14.038173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":741713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.71799999999999997}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.626},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.60499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":151871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.69699999999999995}]},{"osd":2,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.60099999999999998}]},{"osd":3,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57999999999999996}]},{"osd":4,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.38},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Jan 15 14:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":146589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.22}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.308}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.625}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.53500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":741623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.3369999999999997,"5min":6.3369999999999997,"15min":6.3369999999999997},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.0339999999999998,"5min":4.0339999999999998,"15min":4.0339999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.4669999999999996,"5min":6.4669999999999996,"15min":6.4669999999999996},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":19.106000000000002,"5min":19.106000000000002,"15min":19.106000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":6.383,"5min":6.383,"15min":6.383},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":19.045999999999999,"5min":19.045999999999999,"15min":19.045999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.599,"15min":1.599},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":18.984999999999999,"5min":18.984999999999999,"15min":18.984999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":4.0650000000000004,"5min":4.0650000000000004,"15min":4.0650000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.4390000000000001,"5min":6.4390000000000001,"15min":6.4390000000000001},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":6.4119999999999999,"5min":6.4119999999999999,"15min":6.4119999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":19.079000000000001,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.308,"15min":0.308},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34000000000000002}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41199999999999998}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.53400000000000003}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.5}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":152889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"last":0.30499999999999999}]},{"osd":1,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.495},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.45600000000000002}]},{"osd":6,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.442}]},{"osd":7,"last update":"Mon Jan 15 14:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44700000000000001}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.5},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.5869999999999997,"5min":4.5869999999999997,"15min":4.5869999999999997},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.65000000000000002}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.622},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.629},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.69999999999999996}]},{"osd":2,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.0819999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.1899999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.77100000000000002}]},{"osd":7,"last update":"Mon Jan 15 14:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":7.8230000000000004,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.871}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T14:36:04.246 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T14:36:04.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T14:36:04.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.411+0000 7fdab58e1700 1 Processor -- start 2024-01-15T14:36:04.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.411+0000 7fdab58e1700 1 -- start start 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdab58e1700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdab015e110 0x7fdab006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdab58e1700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdab00afb30 0x7fdab006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdab58e1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab00affe0 con 0x7fdab006b530 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdab58e1700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab006d980 con 0x7fdab015e110 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdab58e1700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab00b0260 con 0x7fdab00afb30 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdaaeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdab015e110 0x7fdab006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdaaf7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdab006b530 0x7fdab006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54228/0 (socket says 172.21.15.77:54228) 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.412+0000 7fdaaf7fe700 1 -- 172.21.15.77:0/3445320711 learned_addr learned my addr 172.21.15.77:0/3445320711 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:04.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaae7fc700 1 --2- 172.21.15.77:0/3445320711 >> v2:172.21.15.159:3300/0 conn(0x7fdab00afb30 0x7fdab006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3445320711 >> v2:172.21.15.159:3300/0 conn(0x7fdab00afb30 msgr2=0x7fdab006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaaeffd700 1 --2- 172.21.15.77:0/3445320711 >> v2:172.21.15.159:3300/0 conn(0x7fdab00afb30 0x7fdab006aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3445320711 >> v1:172.21.15.77:6789/0 conn(0x7fdab006b530 legacy=0x7fdab006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaadffb700 1 -- 172.21.15.77:0/3445320711 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576190437 0 0) 0x7fdab00affe0 con 0x7fdab006b530 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3445320711 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab00b0540 con 0x7fdab015e110 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaae7fc700 1 --2- 172.21.15.77:0/3445320711 >> v2:172.21.15.159:3300/0 conn(0x7fdab00afb30 0x7fdab006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaaeffd700 1 --2- 172.21.15.77:0/3445320711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab015e110 0x7fdab006f680 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdab00b3990 tx=0x7fdaa4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bd10e42a4deff7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaadffb700 1 -- 172.21.15.77:0/3445320711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4011830 con 0x7fdab015e110 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaadffb700 1 -- 172.21.15.77:0/3445320711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa4011990 con 0x7fdab015e110 2024-01-15T14:36:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.413+0000 7fdaadffb700 1 -- 172.21.15.77:0/3445320711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4011d20 con 0x7fdab015e110 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 -- 172.21.15.77:0/3445320711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab015e110 msgr2=0x7fdab006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3445320711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab015e110 0x7fdab006f680 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdab00b3990 tx=0x7fdaa4009b20 comp rx=0 tx=0).stop 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 -- 172.21.15.77:0/3445320711 shutdown_connections 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3445320711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab015e110 0x7fdab006f680 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3445320711 >> v2:172.21.15.159:3300/0 conn(0x7fdab00afb30 0x7fdab006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 -- 172.21.15.77:0/3445320711 >> 172.21.15.77:0/3445320711 conn(0x7fdab00b3250 msgr2=0x7fdab00d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 -- 172.21.15.77:0/3445320711 shutdown_connections 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.414+0000 7fdab58e1700 1 -- 172.21.15.77:0/3445320711 wait complete. 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.415+0000 7fdab58e1700 1 Processor -- start 2024-01-15T14:36:04.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.415+0000 7fdab58e1700 1 -- start start 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 0x7fdab015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdab00afb30 0x7fdab0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdab015e110 0x7fdab00744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab01f41b0 con 0x7fdab006b530 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00c5b60 con 0x7fdab00afb30 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab015ae20 con 0x7fdab015e110 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 0x7fdab015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 0x7fdab015a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57594/0 (socket says 172.21.15.77:57594) 2024-01-15T14:36:04.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3508043386 learned_addr learned my addr 172.21.15.77:0/3508043386 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaae7fc700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdab00afb30 0x7fdab0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdab015e110 msgr2=0x7fdab00744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaf7fe700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdab015e110 0x7fdab00744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdab015e110 0x7fdab00744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdab00afb30 msgr2=0x7fdab0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdab00afb30 0x7fdab0073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 -- 172.21.15.77:0/3508043386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa40094a0 con 0x7fdab006b530 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaae7fc700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdab00afb30 0x7fdab0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaf7fe700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdab015e110 0x7fdab00744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdaaeffd700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 0x7fdab015a8e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4014040 tx=0x7fdaa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40038a0 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa401e070 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.416+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab00c5db0 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.417+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4003c60 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.417+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab00c6370 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.418+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab0069020 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.418+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdaa402d450 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.418+0000 7fda97fff700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fda9c06b350 0x7fda9c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.418+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdaa408ac40 con 0x7fdab006b530 2024-01-15T14:36:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.418+0000 7fdaae7fc700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fda9c06b350 0x7fda9c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.419+0000 7fdaae7fc700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fda9c06b350 0x7fda9c06d810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdab00d18f0 tx=0x7fdaa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.423+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdaa4055b40 con 0x7fdab006b530 2024-01-15T14:36:04.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.692+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fdab006cbe0 con 0x7fdab006b530 2024-01-15T14:36:04.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.694+0000 7fda97fff700 1 -- 172.21.15.77:0/3508043386 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7fdaa40150e0 con 0x7fdab006b530 2024-01-15T14:36:04.696 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-01-15T14:36:04.696 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBeQqVl3gmtABAA2JDAPlTu6jT/iTUZPZjfVw== 2024-01-15T14:36:04.696 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-01-15T14:36:04.696 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-01-15T14:36:04.696 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T14:36:04.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fda9c06b350 msgr2=0x7fda9c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fda9c06b350 0x7fda9c06d810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdab00d18f0 tx=0x7fdaa0008040 comp rx=0 tx=0).stop 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 msgr2=0x7fdab015a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 0x7fdab015a8e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4014040 tx=0x7fdaa4004750 comp rx=0 tx=0).stop 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 shutdown_connections 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fda9c06b350 0x7fda9c06d810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdab015e110 0x7fdab00744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab006b530 0x7fdab015a8e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 --2- 172.21.15.77:0/3508043386 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdab00afb30 0x7fdab0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 >> 172.21.15.77:0/3508043386 conn(0x7fdab00b3250 msgr2=0x7fdab00adf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 shutdown_connections 2024-01-15T14:36:04.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.697+0000 7fdab58e1700 1 -- 172.21.15.77:0/3508043386 wait complete. 2024-01-15T14:36:04.700 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-01-15T14:36:04.710 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T14:36:04.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.830+0000 7f7d3e882700 1 Processor -- start 2024-01-15T14:36:04.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.830+0000 7f7d3e882700 1 -- start start 2024-01-15T14:36:04.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d3e882700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d381473a0 0x7f7d3815e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d3e882700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d3806c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d3e882700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3806c9a0 con 0x7f7d3806dc20 2024-01-15T14:36:04.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d3e882700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3815eb00 con 0x7f7d381473a0 2024-01-15T14:36:04.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d3e882700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815ede0 con 0x7f7d38147780 2024-01-15T14:36:04.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d381473a0 0x7f7d3815e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d381473a0 0x7f7d3815e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57596/0 (socket says 172.21.15.77:57596) 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 -- 172.21.15.77:0/2825190357 learned_addr learned my addr 172.21.15.77:0/2825190357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d377fe700 1 --2- 172.21.15.77:0/2825190357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d3806c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 -- 172.21.15.77:0/2825190357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 msgr2=0x7f7d3806c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 --2- 172.21.15.77:0/2825190357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d3806c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 -- 172.21.15.77:0/2825190357 >> v1:172.21.15.159:6789/0 conn(0x7f7d3806dc20 legacy=0x7f7d38073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 -- 172.21.15.77:0/2825190357 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3815f0c0 con 0x7f7d381473a0 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d37fff700 1 --2- 172.21.15.77:0/2825190357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d3815e4b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098f0 tx=0x7f7d2c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdf603cc4d1cd73 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d36ffd700 1 -- 172.21.15.77:0/2825190357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00b8a0 con 0x7f7d381473a0 2024-01-15T14:36:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d36ffd700 1 -- 172.21.15.77:0/2825190357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c00ba00 con 0x7f7d381473a0 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.831+0000 7f7d36ffd700 1 -- 172.21.15.77:0/2825190357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00bd90 con 0x7f7d381473a0 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 -- 172.21.15.77:0/2825190357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 msgr2=0x7f7d3815e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 --2- 172.21.15.77:0/2825190357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d3815e4b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098f0 tx=0x7f7d2c009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 -- 172.21.15.77:0/2825190357 shutdown_connections 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 --2- 172.21.15.77:0/2825190357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d3815e4b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 --2- 172.21.15.77:0/2825190357 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d3806c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 -- 172.21.15.77:0/2825190357 >> 172.21.15.77:0/2825190357 conn(0x7f7d380b3130 msgr2=0x7f7d380d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 -- 172.21.15.77:0/2825190357 shutdown_connections 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.832+0000 7f7d3e882700 1 -- 172.21.15.77:0/2825190357 wait complete. 2024-01-15T14:36:04.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.833+0000 7f7d3e882700 1 Processor -- start 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.833+0000 7f7d3e882700 1 -- start start 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3e882700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3806dc20 0x7f7d3815ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3e882700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d38140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3e882700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d38140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3e882700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38140ea0 con 0x7f7d381473a0 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3e882700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380ada40 con 0x7f7d3806dc20 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3e882700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380add20 con 0x7f7d38147780 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d38140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d38140420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57600/0 (socket says 172.21.15.77:57600) 2024-01-15T14:36:04.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 -- 172.21.15.77:0/826376276 learned_addr learned my addr 172.21.15.77:0/826376276 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d3ce1f700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d38140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d37fff700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3806dc20 0x7f7d3815ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 -- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 msgr2=0x7f7d38140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d38140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 -- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3806dc20 msgr2=0x7f7d3815ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.834+0000 7f7d377fe700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3806dc20 0x7f7d3815ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d377fe700 1 -- 172.21.15.77:0/826376276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c009550 con 0x7f7d381473a0 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d377fe700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d38140420 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800d7d0 tx=0x7f7d2800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280125f0 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28012750 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d380ce280 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28012ae0 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.835+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d381410e0 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.836+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38067f60 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.836+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d28012c40 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.836+0000 7f7d34ff9700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d2406b350 0x7f7d2406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.837+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d28089c10 con 0x7f7d381473a0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.837+0000 7f7d37fff700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d2406b350 0x7f7d2406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:04.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.837+0000 7f7d37fff700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d2406b350 0x7f7d2406d810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009c40 tx=0x7f7d2c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:04.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:04.841+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d28054a80 con 0x7f7d381473a0 2024-01-15T14:36:05.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.076+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3806c240 con 0x7f7d381473a0 2024-01-15T14:36:05.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.076+0000 7f7d34ff9700 1 -- 172.21.15.77:0/826376276 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f7d280580a0 con 0x7f7d381473a0 2024-01-15T14:36:05.078 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:05.078 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4882,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6833/49995002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6832","nonce":49995002},{"type":"v1","addr":"172.21.15.159:6833","nonce":49995002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4885,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.159:6835/2815696160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6834","nonce":2815696160},{"type":"v1","addr":"172.21.15.159:6835","nonce":2815696160}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5082,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3711259103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3711259103},{"type":"v1","addr":"172.21.15.77:6836","nonce":3711259103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T14:35:59.105675+0000","modified":"2024-01-15T14:36:02.124724+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1676979021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":1676979021},{"type":"v1","addr":"172.21.15.77:6837","nonce":1676979021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T14:36:05.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d2406b350 msgr2=0x7f7d2406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d2406b350 0x7f7d2406d810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009c40 tx=0x7f7d2c008030 comp rx=0 tx=0).stop 2024-01-15T14:36:05.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 msgr2=0x7f7d38140420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d38140420 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800d7d0 tx=0x7f7d2800dae0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 shutdown_connections 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7d2406b350 0x7f7d2406d810 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7d38147780 0x7f7d38140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7d3806dc20 0x7f7d3815ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 --2- 172.21.15.77:0/826376276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d381473a0 0x7f7d38140420 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 >> 172.21.15.77:0/826376276 conn(0x7f7d380b3130 msgr2=0x7f7d3815f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 shutdown_connections 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.078+0000 7f7d3e882700 1 -- 172.21.15.77:0/826376276 wait complete. 2024-01-15T14:36:05.081 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 17 2024-01-15T14:36:05.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T14:35:59.105675+0000', 'modified': '2024-01-15T14:36:02.124724+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5088}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5088': {'gid': 5088, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/1676979021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6835', 'nonce': 1676979021}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1676979021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T14:36:05.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T14:36:05.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T14:36:05.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.618+0000 7f7951cb8700 1 Processor -- start 2024-01-15T14:36:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.618+0000 7f7951cb8700 1 -- start start 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f7951cb8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f7951cb8700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f794c0d3a00 0x7f794c0d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f7951cb8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f794c0d0db0 0x7f794c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f7951cb8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c0d3150 con 0x7f794c0d0db0 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f7951cb8700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c0d1240 con 0x7f794c0d3a00 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f7951cb8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c15e5c0 con 0x7f794c15e1e0 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f794bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f794c0d0db0 0x7f794c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f794bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f794c0d0db0 0x7f794c06aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57608/0 (socket says 172.21.15.77:57608) 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.619+0000 7f794bfff700 1 -- 172.21.15.77:0/1590290535 learned_addr learned my addr 172.21.15.77:0/1590290535 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794affd700 1 --2- 172.21.15.77:0/1590290535 >> v2:172.21.15.159:3300/0 conn(0x7f794c0d3a00 0x7f794c0d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1590290535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794bfff700 1 -- 172.21.15.77:0/1590290535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 msgr2=0x7f794c0d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794bfff700 1 --2- 172.21.15.77:0/1590290535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c0d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794bfff700 1 -- 172.21.15.77:0/1590290535 >> v2:172.21.15.159:3300/0 conn(0x7f794c0d3a00 msgr2=0x7f794c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794bfff700 1 --2- 172.21.15.77:0/1590290535 >> v2:172.21.15.159:3300/0 conn(0x7f794c0d3a00 0x7f794c0d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794bfff700 1 -- 172.21.15.77:0/1590290535 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c0d20b0 con 0x7f794c0d0db0 2024-01-15T14:36:05.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794bfff700 1 --2- 172.21.15.77:0/1590290535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d0db0 0x7f794c06aff0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f794c08c1d0 tx=0x7f794400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c6c6e2d13c5bf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794a7fc700 1 -- 172.21.15.77:0/1590290535 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794400e070 con 0x7f794c0d0db0 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794a7fc700 1 -- 172.21.15.77:0/1590290535 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794400b980 con 0x7f794c0d0db0 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f794a7fc700 1 -- 172.21.15.77:0/1590290535 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794400bd30 con 0x7f794c0d0db0 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f7951cb8700 1 -- 172.21.15.77:0/1590290535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d0db0 msgr2=0x7f794c06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.620+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1590290535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d0db0 0x7f794c06aff0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f794c08c1d0 tx=0x7f794400b330 comp rx=0 tx=0).stop 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 -- 172.21.15.77:0/1590290535 shutdown_connections 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1590290535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d0db0 0x7f794c06aff0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1590290535 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c0d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1590290535 >> v2:172.21.15.159:3300/0 conn(0x7f794c0d3a00 0x7f794c0d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 -- 172.21.15.77:0/1590290535 >> 172.21.15.77:0/1590290535 conn(0x7f794c0b5b50 msgr2=0x7f794c0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 -- 172.21.15.77:0/1590290535 shutdown_connections 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.621+0000 7f7951cb8700 1 -- 172.21.15.77:0/1590290535 wait complete. 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.622+0000 7f7951cb8700 1 Processor -- start 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.622+0000 7f7951cb8700 1 -- start start 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f794c0d0db0 0x7f794c156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 0x7f794c154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c15fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c1f7240 con 0x7f794c0d3a00 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c1f7520 con 0x7f794c0d0db0 2024-01-15T14:36:05.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c1f7800 con 0x7f794c15e1e0 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 0x7f794c154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 0x7f794c154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57616/0 (socket says 172.21.15.77:57616) 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 -- 172.21.15.77:0/1130292684 learned_addr learned my addr 172.21.15.77:0/1130292684 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794bfff700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c15fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f794c0d0db0 0x7f794c156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 -- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 msgr2=0x7f794c15fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c15fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 -- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f794c0d0db0 msgr2=0x7f794c156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f794c0d0db0 0x7f794c156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 -- 172.21.15.77:0/1130292684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944009b30 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794bfff700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c15fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f794c0d0db0 0x7f794c156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f794affd700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 0x7f794c154500 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f793c00e810 tx=0x7f793c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7943fff700 1 -- 172.21.15.77:0/1130292684 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c013610 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7943fff700 1 -- 172.21.15.77:0/1130292684 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c013770 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7943fff700 1 -- 172.21.15.77:0/1130292684 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c013b20 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c1f7ae0 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.623+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c154a40 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.625+0000 7f7943fff700 1 -- 172.21.15.77:0/1130292684 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f793c013c80 con 0x7f794c0d3a00 2024-01-15T14:36:05.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.625+0000 7f7943fff700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f792c06b350 0x7f792c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.625+0000 7f7943fff700 1 -- 172.21.15.77:0/1130292684 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f793c089860 con 0x7f794c0d3a00 2024-01-15T14:36:05.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.625+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f792c06b350 0x7f792c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.626+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f792c06b350 0x7f792c06d810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f79340079d0 tx=0x7f7934008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.627+0000 7f795deb4b80 1 Processor -- start 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.627+0000 7f795deb4b80 1 -- start start 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f62acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f607190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f5e41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f795deb4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f557480 con 0x56550f60c410 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f53d770 con 0x56550f5e1dd0 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56550f53b090 con 0x56550f56fce0 2024-01-15T14:36:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f607190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f607190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57628/0 (socket says 172.21.15.77:57628) 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 -- 172.21.15.77:0/3616985032 learned_addr learned my addr 172.21.15.77:0/3616985032 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794bfff700 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f5e41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794b7fe700 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f62acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 -- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 msgr2=0x56550f62acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f62acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 -- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 msgr2=0x56550f5e41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f5e41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794affd700 1 -- 172.21.15.77:0/3616985032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56550f35fb10 con 0x56550f60c410 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.628+0000 7f794bfff700 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f5e41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f794affd700 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f607190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f793c087d90 tx=0x7f793c057db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882e228d7906bca2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f79417fa700 1 -- 172.21.15.77:0/3616985032 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c0583a0 con 0x56550f60c410 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f79417fa700 1 -- 172.21.15.77:0/3616985032 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c05c4f0 con 0x56550f60c410 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f79417fa700 1 -- 172.21.15.77:0/3616985032 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c0606a0 con 0x56550f60c410 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 -- 172.21.15.77:0/3616985032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 msgr2=0x56550f607190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f607190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f793c087d90 tx=0x7f793c057db0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 -- 172.21.15.77:0/3616985032 shutdown_connections 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f62acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f5e41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3616985032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f607190 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 -- 172.21.15.77:0/3616985032 >> 172.21.15.77:0/3616985032 conn(0x56550f511410 msgr2=0x56550f629830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.629+0000 7f795deb4b80 1 -- 172.21.15.77:0/3616985032 shutdown_connections 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.630+0000 7f795deb4b80 1 -- 172.21.15.77:0/3616985032 wait complete. 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.630+0000 7f795deb4b80 1 -- start start 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.630+0000 7f795deb4b80 1 Processor -- start 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f56fce0 0x56550f604d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 0x56550f60bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f60c410 0x56550f639270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f536980 con 0x56550f5e1dd0 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f4d6000 con 0x56550f56fce0 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56550f4d5ca0 con 0x56550f60c410 2024-01-15T14:36:05.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f795deb4b80 10 client.0 _ll_register_callbacks cb 0x56550f339350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 0x56550f60bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 0x56550f60bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57632/0 (socket says 172.21.15.77:57632) 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 -- 172.21.15.77:0/932287057 learned_addr learned my addr 172.21.15.77:0/932287057 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794b7fe700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f56fce0 0x56550f604d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794bfff700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f60c410 0x56550f639270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 -- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f60c410 msgr2=0x56550f639270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f60c410 0x56550f639270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 -- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f56fce0 msgr2=0x56550f604d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f56fce0 0x56550f604d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 -- 172.21.15.77:0/932287057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c154a40 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f794affd700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 0x56550f60bd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f793c060170 tx=0x7f793c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b1ddc353e5e06a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.631+0000 7f792a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f792a7fc700 1 -- 172.21.15.77:0/932287057 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c0606a0 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f792a7fc700 1 -- 172.21.15.77:0/932287057 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f793c057890 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56550f556450 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f792a7fc700 1 -- 172.21.15.77:0/932287057 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f793c057a50 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 10 client.5268 fetch_fsmap learned FSMap version 17 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x56550f4f8290 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f792a7fc700 1 -- 172.21.15.77:0/932287057 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c064070 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f792a7fc700 1 -- 172.21.15.77:0/932287057 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f793c060210 con 0x56550f5e1dd0 2024-01-15T14:36:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 10 client.5268 fetch_fsmap finished waiting for FSMap version 17 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 10 client.5268 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 20 client.5268 populate_metadata read hostname 'smithi077' 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f60ea80 0x56550f60ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 4 client.5268 mds_command: new command op to 5088 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 --> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56550f52f960 con 0x56550f60ea80 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f794b7fe700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f60ea80 0x56550f60ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f792a7fc700 10 client.5268 ms_handle_connect on v2:172.21.15.77:6835/1676979021 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.632+0000 7f794b7fe700 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f60ea80 0x56550f60ee60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ca153715458e7d server_cookie=5ccff2763d301f8e in_seq=0 out_seq=0 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f792a7fc700 1 -- 172.21.15.77:0/932287057 <== mds.0 v2:172.21.15.77:6835/1676979021 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56550f52f960 con 0x56550f60ea80 2024-01-15T14:36:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f792a7fc700 10 client.5268 handle_command_reply: tid=0 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 client.5268 shutdown 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 msgr2=0x56550f60bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 0x56550f60bd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f793c060170 tx=0x7f793c000bc0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 shutdown_connections 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f60c410 0x56550f639270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f60ea80 0x56550f60ee60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f56fce0 0x56550f604d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 --2- 172.21.15.77:0/932287057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f5e1dd0 0x56550f60bd20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f792a7fc700 0 client.5268 ms_handle_reset on v2:172.21.15.77:6835/1676979021 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 >> 172.21.15.77:0/932287057 conn(0x56550f511410 msgr2=0x56550f605e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 shutdown_connections 2024-01-15T14:36:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 1 -- 172.21.15.77:0/932287057 wait complete. 2024-01-15T14:36:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.633+0000 7f795deb4b80 20 client.5268 trim_cache size 0 max 16384 2024-01-15T14:36:05.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.651+0000 7f795deb4b80 1 Processor -- start 2024-01-15T14:36:05.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 -- start start 2024-01-15T14:36:05.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 0x56550f610e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f5e1dd0 0x56550f5e4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f60c410 0x56550f688b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f3b5650 con 0x56550f56fce0 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f605d30 con 0x56550f60c410 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56550f689090 con 0x56550f5e1dd0 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.652+0000 7f794b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 0x56550f610e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 0x56550f610e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57642/0 (socket says 172.21.15.77:57642) 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 -- 172.21.15.77:0/1752192783 learned_addr learned my addr 172.21.15.77:0/1752192783 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794bfff700 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f60c410 0x56550f688b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794affd700 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f5e1dd0 0x56550f5e4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 -- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f5e1dd0 msgr2=0x56550f5e4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f5e1dd0 0x56550f5e4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 -- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f60c410 msgr2=0x56550f688b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f60c410 0x56550f688b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 -- 172.21.15.77:0/1752192783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56550f3b7c80 con 0x56550f56fce0 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f794b7fe700 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 0x56550f610e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f79340070f0 tx=0x7f7934015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd3a2c8617d7cee server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f7941ffb700 1 -- 172.21.15.77:0/1752192783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934028070 con 0x56550f56fce0 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f7941ffb700 1 -- 172.21.15.77:0/1752192783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793401c040 con 0x56550f56fce0 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.653+0000 7f7941ffb700 1 -- 172.21.15.77:0/1752192783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340148a0 con 0x56550f56fce0 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 -- 172.21.15.77:0/1752192783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 msgr2=0x56550f610e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 0x56550f610e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f79340070f0 tx=0x7f7934015000 comp rx=0 tx=0).stop 2024-01-15T14:36:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 -- 172.21.15.77:0/1752192783 shutdown_connections 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f60c410 0x56550f688b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f56fce0 0x56550f610e70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 --2- 172.21.15.77:0/1752192783 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f5e1dd0 0x56550f5e4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 -- 172.21.15.77:0/1752192783 >> 172.21.15.77:0/1752192783 conn(0x56550f511410 msgr2=0x56550f639030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 -- 172.21.15.77:0/1752192783 shutdown_connections 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 -- 172.21.15.77:0/1752192783 wait complete. 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 -- start start 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.654+0000 7f795deb4b80 1 Processor -- start 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f684940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f6871d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f687710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f5df220 con 0x56550f60c410 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550f5e5170 con 0x56550f5e1dd0 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x56550f6850d0 con 0x56550f56fce0 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f795deb4b80 10 client.0 _ll_register_callbacks cb 0x56550f339350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f794bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f687710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f794bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f687710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57658/0 (socket says 172.21.15.77:57658) 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f794bfff700 1 -- 172.21.15.77:0/3796863688 learned_addr learned my addr 172.21.15.77:0/3796863688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f794b7fe700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f684940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.655+0000 7f794affd700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f6871d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794bfff700 1 -- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 msgr2=0x56550f684940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794bfff700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f684940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794bfff700 1 -- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 msgr2=0x56550f6871d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794bfff700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f6871d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794bfff700 1 -- 172.21.15.77:0/3796863688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793400e040 con 0x56550f60c410 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794bfff700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f687710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f794c08c570 tx=0x7f7944004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5482f41770f8849 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f792a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f792a7fc700 1 -- 172.21.15.77:0/3796863688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944004450 con 0x56550f60c410 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f792a7fc700 1 -- 172.21.15.77:0/3796863688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7944003e10 con 0x56550f60c410 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f793c05c4f0 con 0x56550f60c410 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f792a7fc700 1 -- 172.21.15.77:0/3796863688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79440045b0 con 0x56550f60c410 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 10 client.5274 fetch_fsmap learned FSMap version 17 2024-01-15T14:36:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x56550f35dfe0 con 0x56550f60c410 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f792a7fc700 1 -- 172.21.15.77:0/3796863688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c05c4f0 con 0x56550f60c410 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f792a7fc700 1 -- 172.21.15.77:0/3796863688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7944019050 con 0x56550f60c410 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 10 client.5274 fetch_fsmap finished waiting for FSMap version 17 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 10 client.5274 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 20 client.5274 populate_metadata read hostname 'smithi077' 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f62d0f0 0x56550f62d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 4 client.5274 mds_command: new command op to 5088 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 --> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56550f52f680 con 0x56550f62d0f0 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.656+0000 7f794b7fe700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f62d0f0 0x56550f62d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f794b7fe700 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f62d0f0 0x56550f62d4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e2a7299d0eb454b server_cookie=2d72229f8ab65285 in_seq=0 out_seq=0 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f792a7fc700 10 client.5274 ms_handle_connect on v2:172.21.15.77:6835/1676979021 2024-01-15T14:36:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f792a7fc700 1 -- 172.21.15.77:0/3796863688 <== mds.0 v2:172.21.15.77:6835/1676979021 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56550f52f680 con 0x56550f62d0f0 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f792a7fc700 10 client.5274 handle_command_reply: tid=0 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 client.5274 shutdown 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 msgr2=0x56550f687710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f687710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f794c08c570 tx=0x7f7944004b40 comp rx=0 tx=0).stop 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 shutdown_connections 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x56550f56fce0 0x56550f684940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:6835/1676979021,v1:172.21.15.77:6837/1676979021] conn(0x56550f62d0f0 0x56550f62d4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56550f60c410 0x56550f687710 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 --2- 172.21.15.77:0/3796863688 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x56550f5e1dd0 0x56550f6871d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f792a7fc700 0 client.5274 ms_handle_reset on v2:172.21.15.77:6835/1676979021 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 >> 172.21.15.77:0/3796863688 conn(0x56550f511410 msgr2=0x56550f6220e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 shutdown_connections 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 1 -- 172.21.15.77:0/3796863688 wait complete. 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.657+0000 7f795deb4b80 20 client.5274 trim_cache size 0 max 16384 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "a5eae108-559b-4366-a994-4d2898914163", 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5088, 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-01-15T14:36:05.669 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 5.9477426940000004, 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 17, 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 34, 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 34, 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 13.924205545 2024-01-15T14:36:05.670 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:05.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f792c06b350 msgr2=0x7f792c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f792c06b350 0x7f792c06d810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f79340079d0 tx=0x7f7934008040 comp rx=0 tx=0).stop 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 msgr2=0x7f794c154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 0x7f794c154500 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f793c00e810 tx=0x7f793c00ebd0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 shutdown_connections 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f792c06b350 0x7f792c06d810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f794c15e1e0 0x7f794c15fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f794c0d0db0 0x7f794c156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 --2- 172.21.15.77:0/1130292684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f794c0d3a00 0x7f794c154500 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 >> 172.21.15.77:0/1130292684 conn(0x7f794c0b5b50 msgr2=0x7f794c153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 shutdown_connections 2024-01-15T14:36:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.658+0000 7f7951cb8700 1 -- 172.21.15.77:0/1130292684 wait complete. 2024-01-15T14:36:05.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T14:36:05.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.792+0000 7ff2cf8be700 1 Processor -- start 2024-01-15T14:36:05.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cf8be700 1 -- start start 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cf8be700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 0x7ff2c81659f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cf8be700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c806d8b0 con 0x7ff2c815af10 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cf8be700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c806db00 con 0x7ff2c806f720 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cf8be700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c806c110 con 0x7ff2c8072240 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cd65a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff2c815af10 0x7ff2c806ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54234/0 (socket says 172.21.15.77:54234) 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.793+0000 7ff2cd65a700 1 -- 172.21.15.77:0/3294698207 learned_addr learned my addr 172.21.15.77:0/3294698207 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cde5b700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 0x7ff2c81659f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492794243 0 0) 0x7ff2c806d8b0 con 0x7ff2c815af10 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b40035b0 con 0x7ff2c815af10 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2786334321 0 0) 0x7ff2b40035b0 con 0x7ff2c815af10 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 msgr2=0x7ff2c81659f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 0x7ff2c81659f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 >> v1:172.21.15.159:6789/0 conn(0x7ff2c806f720 legacy=0x7ff2c8071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c806dde0 con 0x7ff2c815af10 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7ff2c4001670 con 0x7ff2c815af10 2024-01-15T14:36:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 >> v1:172.21.15.77:6789/0 conn(0x7ff2c815af10 legacy=0x7ff2c806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 0x7ff2b4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b4003fb0 0x7ff2b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4004d90 0x7ff2b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b400bac0 con 0x7ff2b4003fb0 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b400bc30 con 0x7ff2b4004d90 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b400bdf0 con 0x7ff2c8072240 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cce59700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b4003fb0 0x7ff2b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cd65a700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 0x7ff2b4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cde5b700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4004d90 0x7ff2b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cce59700 1 -- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 msgr2=0x7ff2b4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cce59700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8072240 0x7ff2b4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cce59700 1 -- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4004d90 msgr2=0x7ff2b4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.794+0000 7ff2cce59700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4004d90 0x7ff2b4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cce59700 1 -- 172.21.15.77:0/3294698207 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b40035b0 con 0x7ff2b4003fb0 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cde5b700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4004d90 0x7ff2b4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cce59700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b4003fb0 0x7ff2b4004850 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff2c806fb60 tx=0x7ff2b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32b1c847a11647c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0012f0 con 0x7ff2b4003fb0 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc0052f0 con 0x7ff2b4003fb0 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2bbfff700 1 -- 172.21.15.77:0/3294698207 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc006860 con 0x7ff2b4003fb0 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cf8be700 1 -- 172.21.15.77:0/3294698207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b4003fb0 msgr2=0x7ff2b4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/3294698207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b4003fb0 0x7ff2b4004850 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff2c806fb60 tx=0x7ff2b4005880 comp rx=0 tx=0).stop 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cde5b700 1 -- 172.21.15.77:0/3294698207 reap_dead start 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cf8be700 1 -- 172.21.15.77:0/3294698207 shutdown_connections 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cf8be700 1 -- 172.21.15.77:0/3294698207 >> 172.21.15.77:0/3294698207 conn(0x7ff2c80b3190 msgr2=0x7ff2c80d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cf8be700 1 -- 172.21.15.77:0/3294698207 shutdown_connections 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.795+0000 7ff2cf8be700 1 -- 172.21.15.77:0/3294698207 wait complete. 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 Processor -- start 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 -- start start 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4003fb0 0x7ff2c815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8155ab0 0x7ff2c814c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 0x7ff2c80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c815dc20 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80b5890 con 0x7ff2b4003fb0 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cf8be700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c815cfa0 con 0x7ff2c8155ab0 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cde5b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 0x7ff2c80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cde5b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 0x7ff2c80b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57670/0 (socket says 172.21.15.77:57670) 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.797+0000 7ff2cde5b700 1 -- 172.21.15.77:0/4228002984 learned_addr learned my addr 172.21.15.77:0/4228002984 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cd65a700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4003fb0 0x7ff2c815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cce59700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8155ab0 0x7ff2c814c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cde5b700 1 -- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8155ab0 msgr2=0x7ff2c814c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cde5b700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8155ab0 0x7ff2c814c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cde5b700 1 -- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4003fb0 msgr2=0x7ff2c815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cde5b700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4003fb0 0x7ff2c815c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cde5b700 1 -- 172.21.15.77:0/4228002984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c815d280 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cd65a700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4003fb0 0x7ff2c815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cce59700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8155ab0 0x7ff2c814c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cde5b700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 0x7ff2c80b5350 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff2c806e7a0 tx=0x7ff2c00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c000e030 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c0011c60 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c815d560 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0016470 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.798+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c814ca90 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.799+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c815dd60 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.800+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff2c00165d0 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.800+0000 7ff2b9ffb700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff2ac06b350 0x7ff2ac06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.800+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2c000f070 con 0x7ff2c80b4ed0 2024-01-15T14:36:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.800+0000 7ff2cd65a700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff2ac06b350 0x7ff2ac06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.801+0000 7ff2cd65a700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff2ac06b350 0x7ff2ac06d810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4000c00 tx=0x7ff2c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:05.805+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2c815dd60 con 0x7ff2c80b4ed0 2024-01-15T14:36:06.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.038+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff2c815dd60 con 0x7ff2c80b4ed0 2024-01-15T14:36:06.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.039+0000 7ff2b9ffb700 1 -- 172.21.15.77:0/4228002984 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff2c815dd60 con 0x7ff2c80b4ed0 2024-01-15T14:36:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T14:36:06.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff2ac06b350 msgr2=0x7ff2ac06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:06.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff2ac06b350 0x7ff2ac06d810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4000c00 tx=0x7ff2c4009040 comp rx=0 tx=0).stop 2024-01-15T14:36:06.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 msgr2=0x7ff2c80b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 0x7ff2c80b5350 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff2c806e7a0 tx=0x7ff2c00114b0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 shutdown_connections 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7ff2ac06b350 0x7ff2ac06d810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c80b4ed0 0x7ff2c80b5350 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7ff2b4003fb0 0x7ff2c815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 --2- 172.21.15.77:0/4228002984 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7ff2c8155ab0 0x7ff2c814c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 >> 172.21.15.77:0/4228002984 conn(0x7ff2c80b3190 msgr2=0x7ff2c80afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 shutdown_connections 2024-01-15T14:36:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.041+0000 7ff2cf8be700 1 -- 172.21.15.77:0/4228002984 wait complete. 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T14:36:06.054 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T14:36:06.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.055 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-01-15T14:36:06.072 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:36:06.072 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:36:06.072 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:06.073 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42443sec preferred_lft 42443sec 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::944:e5c1:f7e3:6288/64 scope link noprefixroute 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 72:ee:f6:5e:00:6d brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8863:bcff:fea3:4402/64 scope link 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether a2:d0:23:55:e3:05 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a0d0:23ff:fe55:e305/64 scope link 2024-01-15T14:36:06.074 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.075 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T14:36:06.075 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 72:ee:f6:5e:00:6d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T14:36:06.075 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::70ee:f6ff:fe5e:6d/64 scope link 2024-01-15T14:36:06.075 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:06.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.075 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T14:36:06.138 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-15T14:36:06.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.139 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-01-15T14:36:06.199 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-01-15T14:36:06.199 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T14:36:06.199 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T14:36:06.199 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T14:36:06.199 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T14:36:06.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.200 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-15T14:36:06.200 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-15T14:36:06.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T14:36:06.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T14:36:06.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T14:36:06.200 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-15T14:36:06.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.306 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-15T14:36:06.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T14:36:06.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T14:36:06.324 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T14:36:06.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-01-15T14:36:06.391 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:36:06.392 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T14:36:06.449 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T14:36:06.449 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:36:06.449 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T14:36:06.507 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-01-15T14:36:06.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:06.593 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:06.593 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:36:06.612 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:06.612 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:36:06.612 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:36:06.612 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:36:06.613 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:36:06.613 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:36:06.613 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:36:06.613 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=a5eae108-559b-4366-a994-4d2898914163" will pass to kernel 2024-01-15T14:36:06.633 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T14:36:06.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T14:36:06.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:06.661 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:36:06.742 INFO:teuthology.orchestra.run.smithi077.stdout:client4793 2024-01-15T14:36:06.742 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:36:06.742 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:06.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T14:36:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.961+0000 7f1da3cdd700 1 Processor -- start 2024-01-15T14:36:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.961+0000 7f1da3cdd700 1 -- start start 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da3cdd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d9c15aee0 0x7f1d9c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da3cdd700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1d9c0b4340 0x7f1d9c0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da3cdd700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c073520 con 0x7f1d9c0aff00 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da3cdd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c0ae280 con 0x7f1d9c15aee0 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da3cdd700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c0adbc0 con 0x7f1d9c0b4340 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da1a79700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d9c15aee0 0x7f1d9c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da1a79700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d9c15aee0 0x7f1d9c0af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57686/0 (socket says 172.21.15.77:57686) 2024-01-15T14:36:06.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da1a79700 1 -- 172.21.15.77:0/1871613259 learned_addr learned my addr 172.21.15.77:0/1871613259 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.962+0000 7f1da227a700 1 --2- 172.21.15.77:0/1871613259 >> v2:172.21.15.159:3300/0 conn(0x7f1d9c0b4340 0x7f1d9c0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da1a79700 1 -- 172.21.15.77:0/1871613259 >> v2:172.21.15.159:3300/0 conn(0x7f1d9c0b4340 msgr2=0x7f1d9c0ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da1a79700 1 --2- 172.21.15.77:0/1871613259 >> v2:172.21.15.159:3300/0 conn(0x7f1d9c0b4340 0x7f1d9c0ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da1a79700 1 -- 172.21.15.77:0/1871613259 >> v1:172.21.15.159:6789/0 conn(0x7f1d9c0aff00 legacy=0x7f1d9c0b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da1a79700 1 -- 172.21.15.77:0/1871613259 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c0adea0 con 0x7f1d9c15aee0 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da1a79700 1 --2- 172.21.15.77:0/1871613259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c0af9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0af400 tx=0x7f1d8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bceedacad70fba8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da0a77700 1 -- 172.21.15.77:0/1871613259 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c011830 con 0x7f1d9c15aee0 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da0a77700 1 -- 172.21.15.77:0/1871613259 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c011990 con 0x7f1d9c15aee0 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da0a77700 1 -- 172.21.15.77:0/1871613259 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c011d20 con 0x7f1d9c15aee0 2024-01-15T14:36:06.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 -- 172.21.15.77:0/1871613259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c15aee0 msgr2=0x7f1d9c0af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/1871613259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c0af9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0af400 tx=0x7f1d8c009bd0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 -- 172.21.15.77:0/1871613259 shutdown_connections 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/1871613259 >> v2:172.21.15.159:3300/0 conn(0x7f1d9c0b4340 0x7f1d9c0ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/1871613259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c0af9c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 -- 172.21.15.77:0/1871613259 >> 172.21.15.77:0/1871613259 conn(0x7f1d9c0b5640 msgr2=0x7f1d9c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 -- 172.21.15.77:0/1871613259 shutdown_connections 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.963+0000 7f1da3cdd700 1 -- 172.21.15.77:0/1871613259 wait complete. 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 Processor -- start 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 -- start start 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d9c0aff00 0x7f1d9c09a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:06.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 0x7f1d9c154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c09baf0 con 0x7f1d9c0b4340 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c09adc0 con 0x7f1d9c15aee0 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da3cdd700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1548b0 con 0x7f1d9c0aff00 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da1278700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 0x7f1d9c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da1278700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 0x7f1d9c154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57700/0 (socket says 172.21.15.77:57700) 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.965+0000 7f1da1278700 1 -- 172.21.15.77:0/3242503171 learned_addr learned my addr 172.21.15.77:0/3242503171 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1a79700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d9c0aff00 0x7f1d9c09a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da227a700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1278700 1 -- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d9c0aff00 msgr2=0x7f1d9c09a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1278700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d9c0aff00 0x7f1d9c09a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1278700 1 -- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d9c15aee0 msgr2=0x7f1d9c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1278700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c06d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1278700 1 -- 172.21.15.77:0/3242503171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c009550 con 0x7f1d9c0b4340 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da227a700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da1278700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 0x7f1d9c154370 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800c450 tx=0x7f1d9800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d980052d0 con 0x7f1d9c0b4340 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d98005430 con 0x7f1d9c0b4340 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c154b50 con 0x7f1d9c0b4340 2024-01-15T14:36:06.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d980038c0 con 0x7f1d9c0b4340 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.966+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c09b2f0 con 0x7f1d9c0b4340 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.967+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c067f60 con 0x7f1d9c0b4340 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.967+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1d98003a20 con 0x7f1d9c0b4340 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.968+0000 7f1d927fc700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1d8006b350 0x7f1d8006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.968+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d98088fa0 con 0x7f1d9c0b4340 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.968+0000 7f1da1a79700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1d8006b350 0x7f1d8006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:06.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.968+0000 7f1da1a79700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1d8006b350 0x7f1d8006d810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009c10 tx=0x7f1d8c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:06.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:06.972+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d980bbe90 con 0x7f1d9c0b4340 2024-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.211+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1d9c072740 con 0x7f1d9c0b4340 2024-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.212+0000 7f1d927fc700 1 -- 172.21.15.77:0/3242503171 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1d98017070 con 0x7f1d9c0b4340 2024-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1d8006b350 msgr2=0x7f1d8006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1d8006b350 0x7f1d8006d810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009c10 tx=0x7f1d8c008030 comp rx=0 tx=0).stop 2024-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 msgr2=0x7f1d9c154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 0x7f1d9c154370 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800c450 tx=0x7f1d9800c810 comp rx=0 tx=0).stop 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 shutdown_connections 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1d8006b350 0x7f1d8006d810 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1d9c0aff00 0x7f1d9c09a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1d9c15aee0 0x7f1d9c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 --2- 172.21.15.77:0/3242503171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d9c0b4340 0x7f1d9c154370 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 >> 172.21.15.77:0/3242503171 conn(0x7f1d9c0b5640 msgr2=0x7f1d9c0b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 shutdown_connections 2024-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:07.214+0000 7f1da3cdd700 1 -- 172.21.15.77:0/3242503171 wait complete. 2024-01-15T14:36:07.227 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-15T14:36:07.227 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T14:36:07.227 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi159.front.sepia.ceph.com 2024-01-15T14:36:07.227 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T14:36:07.227 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T14:36:07.227 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-15T14:36:07.228 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-15T14:36:07.228 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T14:36:07.228 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T14:36:07.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.228 DEBUG:teuthology.orchestra.run.smithi159:> ip addr 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 ::1/128 scope host 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:36:07.251 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c6 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f0 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:d9:d8:c7 brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: altname enp7s0f1 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4e brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 0c:c4:7a:88:72:4f brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: inet 172.21.15.159/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T14:36:07.252 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft 42445sec preferred_lft 42445sec 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::6b97:3ef3:99d1:2a4/64 scope link noprefixroute 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 82:0c:ca:d3:20:ef brd ff:ff:ff:ff:ff:ff 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::b8cb:fff:fe98:f453/64 scope link 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether a6:ae:a0:c4:3e:57 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T14:36:07.253 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::a4ae:a0ff:fec4:3e57/64 scope link 2024-01-15T14:36:07.254 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.254 INFO:teuthology.orchestra.run.smithi159.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T14:36:07.254 INFO:teuthology.orchestra.run.smithi159.stdout: link/ether 82:0c:ca:d3:20:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T14:36:07.254 INFO:teuthology.orchestra.run.smithi159.stdout: inet6 fe80::800c:caff:fed3:20ef/64 scope link 2024-01-15T14:36:07.254 INFO:teuthology.orchestra.run.smithi159.stdout: valid_lft forever preferred_lft forever 2024-01-15T14:36:07.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.254 DEBUG:teuthology.orchestra.run.smithi159:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T14:36:07.323 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-01-15T14:36:07.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.325 DEBUG:teuthology.orchestra.run.smithi159:> route 2024-01-15T14:36:07.387 INFO:teuthology.orchestra.run.smithi159.stdout:Kernel IP routing table 2024-01-15T14:36:07.388 INFO:teuthology.orchestra.run.smithi159.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T14:36:07.388 INFO:teuthology.orchestra.run.smithi159.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T14:36:07.388 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T14:36:07.388 INFO:teuthology.orchestra.run.smithi159.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T14:36:07.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.388 DEBUG:teuthology.orchestra.run.smithi159:> (cd / && exec bash -c ' 2024-01-15T14:36:07.389 DEBUG:teuthology.orchestra.run.smithi159:> set -e 2024-01-15T14:36:07.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T14:36:07.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T14:36:07.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T14:36:07.389 DEBUG:teuthology.orchestra.run.smithi159:> ') 2024-01-15T14:36:07.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.500 DEBUG:teuthology.orchestra.run.smithi159:> ip netns list 2024-01-15T14:36:07.518 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T14:36:07.518 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T14:36:07.519 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T14:36:07.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.519 DEBUG:teuthology.orchestra.run.smithi159:> sudo ip link set brx.0 up 2024-01-15T14:36:07.587 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:36:07.587 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T14:36:07.644 INFO:teuthology.orchestra.run.smithi159.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T14:36:07.644 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T14:36:07.644 DEBUG:teuthology.orchestra.run.smithi159:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T14:36:07.701 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-01-15T14:36:07.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:07.787 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:07.788 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:36:07.807 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:07.807 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:36:07.807 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:36:07.807 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 1@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:36:07.807 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:36:07.807 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:36:07.808 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:36:07.808 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.159:3300,172.21.15.159:3301" 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with new device syntax: 1@a5eae108-559b-4366-a994-4d2898914163.cephfs=/ 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.77:3300/172.21.15.159:3300/172.21.15.159:3301" will pass to kernel 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: v2 mount support check returned 2 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: kernel does not support v2 syntax 2024-01-15T14:36:07.826 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-15T14:36:07.827 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790" 2024-01-15T14:36:07.827 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.159:6789,172.21.15.159:6790:/ 2024-01-15T14:36:07.827 INFO:teuthology.orchestra.run.smithi159.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=a5eae108-559b-4366-a994-4d2898914163" will pass to kernel 2024-01-15T14:36:07.828 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-15T14:36:07.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.829 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T14:36:07.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:07.857 DEBUG:teuthology.orchestra.run.smithi159:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-15T14:36:07.930 INFO:teuthology.orchestra.run.smithi159.stdout:client4799 2024-01-15T14:36:07.931 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-01-15T14:36:07.931 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:07.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T14:36:08.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.126+0000 7fa2bf9c3700 1 Processor -- start 2024-01-15T14:36:08.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.126+0000 7fa2bf9c3700 1 -- start start 2024-01-15T14:36:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bf9c3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b81622b0 0x7fa2b806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bf9c3700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa2b8162820 0x7fa2b806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bf9c3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b806bed0 con 0x7fa2b815df20 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bf9c3700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b806cfe0 con 0x7fa2b8162820 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bf9c3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b806bba0 con 0x7fa2b81622b0 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bd75f700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa2b815df20 0x7fa2b8161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54246/0 (socket says 172.21.15.77:54246) 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.127+0000 7fa2bd75f700 1 -- 172.21.15.77:0/837150736 learned_addr learned my addr 172.21.15.77:0/837150736 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bcf5e700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b81622b0 0x7fa2b806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bdf60700 1 --2- 172.21.15.77:0/837150736 >> v2:172.21.15.159:3300/0 conn(0x7fa2b8162820 0x7fa2b806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2affff700 1 -- 172.21.15.77:0/837150736 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215313778 0 0) 0x7fa2b806bed0 con 0x7fa2b815df20 2024-01-15T14:36:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2affff700 1 -- 172.21.15.77:0/837150736 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a40035b0 con 0x7fa2b815df20 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bdf60700 1 -- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b81622b0 msgr2=0x7fa2b806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bdf60700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b81622b0 0x7fa2b806c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2affff700 1 -- 172.21.15.77:0/837150736 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1883156630 0 0) 0x7fa2a40035b0 con 0x7fa2b815df20 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bdf60700 1 -- 172.21.15.77:0/837150736 >> v1:172.21.15.77:6789/0 conn(0x7fa2b815df20 legacy=0x7fa2b8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bdf60700 1 -- 172.21.15.77:0/837150736 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b8070060 con 0x7fa2b8162820 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bcf5e700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b81622b0 0x7fa2b806c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2bdf60700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b806caa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009d90 tx=0x7fa2b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b01c8e838557740 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2affff700 1 -- 172.21.15.77:0/837150736 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000e070 con 0x7fa2b8162820 2024-01-15T14:36:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2affff700 1 -- 172.21.15.77:0/837150736 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b0003a20 con 0x7fa2b8162820 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.128+0000 7fa2affff700 1 -- 172.21.15.77:0/837150736 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4001670 con 0x7fa2b8162820 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 msgr2=0x7fa2b806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b806caa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009d90 tx=0x7fa2b000b330 comp rx=0 tx=0).stop 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/837150736 shutdown_connections 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b806caa0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/837150736 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b81622b0 0x7fa2b806c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/837150736 >> 172.21.15.77:0/837150736 conn(0x7fa2b806dc80 msgr2=0x7fa2b809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/837150736 shutdown_connections 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.129+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/837150736 wait complete. 2024-01-15T14:36:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.130+0000 7fa2bf9c3700 1 Processor -- start 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.130+0000 7fa2bf9c3700 1 -- start start 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bf9c3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b815df20 0x7fa2b81f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bf9c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 0x7fa2b80b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bf9c3700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b80b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bf9c3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b8160770 con 0x7fa2b81622b0 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bf9c3700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b815f6f0 con 0x7fa2b8162820 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bf9c3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b815f9b0 con 0x7fa2b815df20 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 0x7fa2b80b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 0x7fa2b80b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57710/0 (socket says 172.21.15.77:57710) 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 -- 172.21.15.77:0/48564681 learned_addr learned my addr 172.21.15.77:0/48564681 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bdf60700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b80b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bd75f700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b815df20 0x7fa2b81f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 -- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b815df20 msgr2=0x7fa2b81f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b815df20 0x7fa2b81f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 -- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 msgr2=0x7fa2b80b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b80b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.131+0000 7fa2bcf5e700 1 -- 172.21.15.77:0/48564681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a80094a0 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.132+0000 7fa2bcf5e700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 0x7fa2b80b1f60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800ebe0 tx=0x7fa2a800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.132+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8009760 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.132+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a800c850 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.132+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b0009a70 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.132+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a800cbe0 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.132+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b815ff70 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.133+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b8067f60 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.133+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa2a800cd40 con 0x7fa2b81622b0 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.134+0000 7fa2adffb700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa29c06b350 0x7fa29c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.134+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2a8089c40 con 0x7fa2b81622b0 2024-01-15T14:36:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.134+0000 7fa2bd75f700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa29c06b350 0x7fa29c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.134+0000 7fa2bd75f700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa29c06b350 0x7fa29c06d810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4000f60 tx=0x7fa2b4001e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.138+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2a8054a30 con 0x7fa2b81622b0 2024-01-15T14:36:08.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:08.373+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa2b80b7d90 con 0x7fa2b81622b0 2024-01-15T14:36:09.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.183+0000 7fa2adffb700 1 -- 172.21.15.77:0/48564681 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa2a8058050 con 0x7fa2b81622b0 2024-01-15T14:36:09.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa29c06b350 msgr2=0x7fa29c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa29c06b350 0x7fa29c06d810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4000f60 tx=0x7fa2b4001e00 comp rx=0 tx=0).stop 2024-01-15T14:36:09.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 msgr2=0x7fa2b80b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 0x7fa2b80b1f60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800ebe0 tx=0x7fa2a800eef0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 shutdown_connections 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa29c06b350 0x7fa29c06d810 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa2b815df20 0x7fa2b81f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa2b8162820 0x7fa2b80b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 --2- 172.21.15.77:0/48564681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2b81622b0 0x7fa2b80b1f60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 >> 172.21.15.77:0/48564681 conn(0x7fa2b806dc80 msgr2=0x7fa2b806ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 shutdown_connections 2024-01-15T14:36:09.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.186+0000 7fa2bf9c3700 1 -- 172.21.15.77:0/48564681 wait complete. 2024-01-15T14:36:09.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T14:36:09.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 Processor -- start 2024-01-15T14:36:09.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 -- start start 2024-01-15T14:36:09.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a6015bd30 0x7f9a600cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a60161bf0 con 0x7f9a60162920 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a600cff40 con 0x7f9a6015bd30 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a6572c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6015f450 con 0x7f9a6015edf0 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a5effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a6015bd30 0x7f9a600cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a5e7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9a60162920 0x7f9a6015e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41292/0 (socket says 172.21.15.77:41292) 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.327+0000 7f9a5e7fc700 1 -- 172.21.15.77:0/2083945223 learned_addr learned my addr 172.21.15.77:0/2083945223 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5f7fe700 1 --2- 172.21.15.77:0/2083945223 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5effd700 1 -- 172.21.15.77:0/2083945223 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 msgr2=0x7f9a600cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5effd700 1 --2- 172.21.15.77:0/2083945223 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5effd700 1 -- 172.21.15.77:0/2083945223 >> v1:172.21.15.77:6789/0 conn(0x7f9a60162920 legacy=0x7f9a6015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5effd700 1 -- 172.21.15.77:0/2083945223 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60161e40 con 0x7f9a6015bd30 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5dffb700 1 -- 172.21.15.77:0/2083945223 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232511319 0 0) 0x7f9a60161bf0 con 0x7f9a60162920 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5effd700 1 --2- 172.21.15.77:0/2083945223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a6015bd30 0x7f9a600cf4c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009840 tx=0x7f9a50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc901948beb1cec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5dffb700 1 -- 172.21.15.77:0/2083945223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50011830 con 0x7f9a6015bd30 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5dffb700 1 -- 172.21.15.77:0/2083945223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a50011990 con 0x7f9a6015bd30 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.328+0000 7f9a5dffb700 1 -- 172.21.15.77:0/2083945223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50011d20 con 0x7f9a6015bd30 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 -- 172.21.15.77:0/2083945223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a6015bd30 msgr2=0x7f9a600cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 --2- 172.21.15.77:0/2083945223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a6015bd30 0x7f9a600cf4c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009840 tx=0x7f9a50009b50 comp rx=0 tx=0).stop 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 -- 172.21.15.77:0/2083945223 shutdown_connections 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 --2- 172.21.15.77:0/2083945223 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 --2- 172.21.15.77:0/2083945223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a6015bd30 0x7f9a600cf4c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 -- 172.21.15.77:0/2083945223 >> 172.21.15.77:0/2083945223 conn(0x7f9a60072710 msgr2=0x7f9a60072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 -- 172.21.15.77:0/2083945223 shutdown_connections 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.329+0000 7f9a6572c700 1 -- 172.21.15.77:0/2083945223 wait complete. 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.330+0000 7f9a6572c700 1 Processor -- start 2024-01-15T14:36:09.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.330+0000 7f9a6572c700 1 -- start start 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a6572c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a6015bd30 0x7f9a600b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a6572c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a6572c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 0x7f9a601603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a6572c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b5f70 con 0x7f9a60162920 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a6572c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600ae060 con 0x7f9a6015bd30 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a6572c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b11c0 con 0x7f9a6015edf0 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a5f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 0x7f9a601603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a5f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 0x7f9a601603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37028/0 (socket says 172.21.15.77:37028) 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.331+0000 7f9a5f7fe700 1 -- 172.21.15.77:0/799890661 learned_addr learned my addr 172.21.15.77:0/799890661 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5effd700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a6015bd30 0x7f9a600b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5e7fc700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5f7fe700 1 -- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 msgr2=0x7f9a600adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5f7fe700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5f7fe700 1 -- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a6015bd30 msgr2=0x7f9a600b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5f7fe700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a6015bd30 0x7f9a600b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5f7fe700 1 -- 172.21.15.77:0/799890661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a500094a0 con 0x7f9a60162920 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5effd700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a6015bd30 0x7f9a600b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:09.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5e7fc700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a5f7fe700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 0x7f9a601603c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a58009dd0 tx=0x7f9a5800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5800cd40 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5800f040 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a600b14a0 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580146c0 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.332+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a600b1a60 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.333+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a600ccb20 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.333+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9a58014820 con 0x7f9a60162920 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.334+0000 7f9a477fe700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9a4c06b350 0x7f9a4c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.334+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a5808a7b0 con 0x7f9a60162920 2024-01-15T14:36:09.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.334+0000 7f9a5effd700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9a4c06b350 0x7f9a4c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.334+0000 7f9a5effd700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9a4c06b350 0x7f9a4c06d810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9a500090b0 tx=0x7f9a50008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:09.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.338+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a58055630 con 0x7f9a60162920 2024-01-15T14:36:09.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.557+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a600cf0a0 con 0x7f9a4c06b350 2024-01-15T14:36:09.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.559+0000 7f9a477fe700 1 -- 172.21.15.77:0/799890661 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9a600cf0a0 con 0x7f9a4c06b350 2024-01-15T14:36:09.560 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-01-15T14:36:09.560 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:09.560 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephfs" 2024-01-15T14:36:09.561 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:09.561 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-01-15T14:36:09.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9a4c06b350 msgr2=0x7f9a4c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9a4c06b350 0x7f9a4c06d810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9a500090b0 tx=0x7f9a50008030 comp rx=0 tx=0).stop 2024-01-15T14:36:09.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 msgr2=0x7f9a601603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 0x7f9a601603c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a58009dd0 tx=0x7f9a5800c3a0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 shutdown_connections 2024-01-15T14:36:09.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f9a4c06b350 0x7f9a4c06d810 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a60162920 0x7f9a601603c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f9a6015bd30 0x7f9a600b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 --2- 172.21.15.77:0/799890661 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f9a6015edf0 0x7f9a600adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 >> 172.21.15.77:0/799890661 conn(0x7f9a60072710 msgr2=0x7f9a600caed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:09.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 shutdown_connections 2024-01-15T14:36:09.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.561+0000 7f9a6572c700 1 -- 172.21.15.77:0/799890661 wait complete. 2024-01-15T14:36:09.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T14:36:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.698+0000 7f68edb8c700 1 Processor -- start 2024-01-15T14:36:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.698+0000 7f68edb8c700 1 -- start start 2024-01-15T14:36:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68edb8c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68edb8c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80b42a0 con 0x7f68e815e280 2024-01-15T14:36:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68edb8c700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80af900 con 0x7f68e80b5220 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68edb8c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8074d30 con 0x7f68e8073f90 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68e77fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f68e815e280 0x7f68e80b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41296/0 (socket says 172.21.15.77:41296) 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68e77fe700 1 -- 172.21.15.77:0/2042192883 learned_addr learned my addr 172.21.15.77:0/2042192883 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68e7fff700 1 --2- 172.21.15.77:0/2042192883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857418263 0 0) 0x7f68d0001110 con 0x7f68e815e280 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.699+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d40035b0 con 0x7f68e815e280 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957198801 0 0) 0x7f68e80af900 con 0x7f68e80b5220 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d0001110 con 0x7f68e80b5220 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1582478941 0 0) 0x7f68d40035b0 con 0x7f68e815e280 2024-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e7fff700 1 -- 172.21.15.77:0/2042192883 >> v1:172.21.15.159:6789/0 conn(0x7f68e80b5220 legacy=0x7f68e806e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e7fff700 1 -- 172.21.15.77:0/2042192883 >> v1:172.21.15.77:6789/0 conn(0x7f68e815e280 legacy=0x7f68e80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e7fff700 1 -- 172.21.15.77:0/2042192883 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e8074490 con 0x7f68e8073f90 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e7fff700 1 --2- 172.21.15.77:0/2042192883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e80b3d60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009dc0 tx=0x7f68e000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d9fe81d44024c32 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000e070 con 0x7f68e8073f90 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e0003a20 con 0x7f68e8073f90 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.700+0000 7f68e67fc700 1 -- 172.21.15.77:0/2042192883 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0001800 con 0x7f68e8073f90 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 -- 172.21.15.77:0/2042192883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 msgr2=0x7f68e80b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2042192883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e80b3d60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009dc0 tx=0x7f68e000b330 comp rx=0 tx=0).stop 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 -- 172.21.15.77:0/2042192883 shutdown_connections 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2042192883 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e80b3d60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 -- 172.21.15.77:0/2042192883 >> 172.21.15.77:0/2042192883 conn(0x7f68e8072b20 msgr2=0x7f68e806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 -- 172.21.15.77:0/2042192883 shutdown_connections 2024-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.701+0000 7f68edb8c700 1 -- 172.21.15.77:0/2042192883 wait complete. 2024-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.702+0000 7f68edb8c700 1 Processor -- start 2024-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.702+0000 7f68edb8c700 1 -- start start 2024-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68edb8c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68edb8c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 0x7f68e81476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68edb8c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68e815b6e0 0x7f68e815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68edb8c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815e6e0 con 0x7f68e80b5220 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68edb8c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815c0b0 con 0x7f68e815b6e0 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68edb8c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815e930 con 0x7f68e8073f90 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 0x7f68e81476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68e6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 0x7f68e81476f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37040/0 (socket says 172.21.15.77:37040) 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.703+0000 7f68e6ffd700 1 -- 172.21.15.77:0/2216117765 learned_addr learned my addr 172.21.15.77:0/2216117765 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e7fff700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68e815b6e0 0x7f68e815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e77fe700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e6ffd700 1 -- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 msgr2=0x7f68e815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e6ffd700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e815f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e6ffd700 1 -- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68e815b6e0 msgr2=0x7f68e815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e6ffd700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68e815b6e0 0x7f68e815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e6ffd700 1 -- 172.21.15.77:0/2216117765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0009a70 con 0x7f68e80b5220 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e7fff700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68e815b6e0 0x7f68e815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68e6ffd700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 0x7f68e81476f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f68d800c520 tx=0x7f68d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80052d0 con 0x7f68e80b5220 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d8005430 con 0x7f68e80b5220 2024-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e815ec10 con 0x7f68e80b5220 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80038c0 con 0x7f68e80b5220 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.704+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e815f1d0 con 0x7f68e80b5220 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.705+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e8069020 con 0x7f68e80b5220 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.705+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f68d8003a20 con 0x7f68e80b5220 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.706+0000 7f68dffff700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68c806b350 0x7f68c806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.706+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68d8089140 con 0x7f68e80b5220 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.706+0000 7f68e77fe700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68c806b350 0x7f68c806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.706+0000 7f68e77fe700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68c806b350 0x7f68c806d810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68d00079b0 tx=0x7f68d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.710+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68d8053fb0 con 0x7f68e80b5220 2024-01-15T14:36:09.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.901+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f68e815d070 con 0x7f68e80b5220 2024-01-15T14:36:09.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.902+0000 7f68dffff700 1 -- 172.21.15.77:0/2216117765 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f68d80575d0 con 0x7f68e80b5220 2024-01-15T14:36:09.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68c806b350 msgr2=0x7f68c806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68c806b350 0x7f68c806d810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68d00079b0 tx=0x7f68d0009040 comp rx=0 tx=0).stop 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 msgr2=0x7f68e81476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 0x7f68e81476f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f68d800c520 tx=0x7f68d800c8e0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 shutdown_connections 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f68c806b350 0x7f68c806d810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f68e8073f90 0x7f68e815f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f68e815b6e0 0x7f68e815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 --2- 172.21.15.77:0/2216117765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e80b5220 0x7f68e81476f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 >> 172.21.15.77:0/2216117765 conn(0x7f68e8072b20 msgr2=0x7f68e815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 shutdown_connections 2024-01-15T14:36:09.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:09.904+0000 7f68edb8c700 1 -- 172.21.15.77:0/2216117765 wait complete. 2024-01-15T14:36:09.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T14:36:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.042+0000 7f425758e700 1 Processor -- start 2024-01-15T14:36:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.042+0000 7f425758e700 1 -- start start 2024-01-15T14:36:10.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425758e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f425015dfa0 0x7f42500cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425758e700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f42500b1260 0x7f42500ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425758e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425006b460 con 0x7f42500d1b00 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425758e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42500cee20 con 0x7f425015dfa0 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425758e700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42500cf100 con 0x7f42500b1260 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425532a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f425015dfa0 0x7f42500cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4255b2b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f42500d1b00 0x7f42500d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41310/0 (socket says 172.21.15.77:41310) 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4255b2b700 1 -- 172.21.15.77:0/2774429765 learned_addr learned my addr 172.21.15.77:0/2774429765 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4254b29700 1 --2- 172.21.15.77:0/2774429765 >> v2:172.21.15.159:3300/0 conn(0x7f42500b1260 0x7f42500ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425532a700 1 -- 172.21.15.77:0/2774429765 >> v2:172.21.15.159:3300/0 conn(0x7f42500b1260 msgr2=0x7f42500ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425532a700 1 --2- 172.21.15.77:0/2774429765 >> v2:172.21.15.159:3300/0 conn(0x7f42500b1260 0x7f42500ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425532a700 1 -- 172.21.15.77:0/2774429765 >> v1:172.21.15.77:6789/0 conn(0x7f42500d1b00 legacy=0x7f42500d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425532a700 1 -- 172.21.15.77:0/2774429765 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425006b6d0 con 0x7f425015dfa0 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4247fff700 1 -- 172.21.15.77:0/2774429765 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005820479 0 0) 0x7f425006b460 con 0x7f42500d1b00 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4254b29700 1 --2- 172.21.15.77:0/2774429765 >> v2:172.21.15.159:3300/0 conn(0x7f42500b1260 0x7f42500ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f425532a700 1 --2- 172.21.15.77:0/2774429765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425015dfa0 0x7f42500cfd90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f425009b820 tx=0x7f424c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0ef25a76591d64 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4247fff700 1 -- 172.21.15.77:0/2774429765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c011830 con 0x7f425015dfa0 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4247fff700 1 -- 172.21.15.77:0/2774429765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c011990 con 0x7f425015dfa0 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.043+0000 7f4247fff700 1 -- 172.21.15.77:0/2774429765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c011d20 con 0x7f425015dfa0 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 -- 172.21.15.77:0/2774429765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425015dfa0 msgr2=0x7f42500cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 --2- 172.21.15.77:0/2774429765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425015dfa0 0x7f42500cfd90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f425009b820 tx=0x7f424c009b20 comp rx=0 tx=0).stop 2024-01-15T14:36:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 -- 172.21.15.77:0/2774429765 shutdown_connections 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 --2- 172.21.15.77:0/2774429765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425015dfa0 0x7f42500cfd90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 --2- 172.21.15.77:0/2774429765 >> v2:172.21.15.159:3300/0 conn(0x7f42500b1260 0x7f42500ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 -- 172.21.15.77:0/2774429765 >> 172.21.15.77:0/2774429765 conn(0x7f425009aa00 msgr2=0x7f425009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 -- 172.21.15.77:0/2774429765 shutdown_connections 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.044+0000 7f425758e700 1 -- 172.21.15.77:0/2774429765 wait complete. 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.045+0000 7f425758e700 1 Processor -- start 2024-01-15T14:36:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.045+0000 7f425758e700 1 -- start start 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425758e700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42500b1260 0x7f42501583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425758e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 0x7f42500b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425758e700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f425015dfa0 0x7f425006baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425758e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42500b4a50 con 0x7f42500d1b00 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425758e700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42500b4420 con 0x7f42500b1260 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425758e700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42500b4d10 con 0x7f425015dfa0 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 0x7f42500b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 0x7f42500b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37056/0 (socket says 172.21.15.77:37056) 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 -- 172.21.15.77:0/2579295506 learned_addr learned my addr 172.21.15.77:0/2579295506 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f425532a700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42500b1260 0x7f42501583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4255b2b700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f425015dfa0 0x7f425006baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 -- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f425015dfa0 msgr2=0x7f425006baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f425015dfa0 0x7f425006baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 -- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42500b1260 msgr2=0x7f42501583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42500b1260 0x7f42501583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 -- 172.21.15.77:0/2579295506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c0094a0 con 0x7f42500d1b00 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.046+0000 7f4254b29700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 0x7f42500b3a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f424000c7f0 tx=0x7f424000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.047+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240004220 con 0x7f42500d1b00 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.047+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4240004380 con 0x7f42500d1b00 2024-01-15T14:36:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.047+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42500b4ff0 con 0x7f42500d1b00 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.047+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42400036a0 con 0x7f42500d1b00 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.047+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42501588f0 con 0x7f42500d1b00 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.048+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4250067f60 con 0x7f42500d1b00 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.048+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4240003800 con 0x7f42500d1b00 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.049+0000 7f4245ffb700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f423406b350 0x7f423406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.049+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4240089770 con 0x7f42500d1b00 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.049+0000 7f425532a700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f423406b350 0x7f423406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.049+0000 7f425532a700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f423406b350 0x7f423406d810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f424c0090b0 tx=0x7f424c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.053+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42400545e0 con 0x7f42500d1b00 2024-01-15T14:36:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.276+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f42500ce9c0 con 0x7f423406b350 2024-01-15T14:36:10.281 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:10.280+0000 7feb9d21d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:10.281 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:10.280+0000 7feb9d21d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:10.282 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:10.280+0000 7feb9d21d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:10.282 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:10.280+0000 7feb9d21d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:10.282 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:10.280+0000 7feb9d21d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:10.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.315+0000 7f4245ffb700 1 -- 172.21.15.77:0/2579295506 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f424c011c00 con 0x7f423406b350 2024-01-15T14:36:10.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.317+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f423406b350 msgr2=0x7f423406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f423406b350 0x7f423406d810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f424c0090b0 tx=0x7f424c008030 comp rx=0 tx=0).stop 2024-01-15T14:36:10.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 msgr2=0x7f42500b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 0x7f42500b3a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f424000c7f0 tx=0x7f424000cbb0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 shutdown_connections 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f423406b350 0x7f423406d810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f425015dfa0 0x7f425006baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42500b1260 0x7f42501583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 --2- 172.21.15.77:0/2579295506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42500d1b00 0x7f42500b3a40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 >> 172.21.15.77:0/2579295506 conn(0x7f425009aa00 msgr2=0x7f42500cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 shutdown_connections 2024-01-15T14:36:10.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.318+0000 7f425758e700 1 -- 172.21.15.77:0/2579295506 wait complete. 2024-01-15T14:36:10.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T14:36:10.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.454+0000 7efe087e1700 1 Processor -- start 2024-01-15T14:36:10.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.455+0000 7efe087e1700 1 -- start start 2024-01-15T14:36:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.455+0000 7efe087e1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe000b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.455+0000 7efe087e1700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe000d03a0 0x7efe0015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.455+0000 7efe087e1700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe000b3860 con 0x7efe0015f700 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.455+0000 7efe087e1700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0015fb90 con 0x7efe000d03a0 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe087e1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe000b2520 con 0x7efe000cffc0 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe0657d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe000b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe06d7e700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efe0015f700 0x7efe0009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:57214/0 (socket says 172.21.15.77:57214) 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe0657d700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe000b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:49894/0 (socket says 172.21.15.77:49894) 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe05d7c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7efe000d03a0 0x7efe0015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe06d7e700 1 -- 172.21.15.77:0/2578869152 learned_addr learned my addr 172.21.15.77:0/2578869152 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe0657d700 1 -- 172.21.15.77:0/2578869152 >> v2:172.21.15.159:3300/0 conn(0x7efe000d03a0 msgr2=0x7efe0015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe0657d700 1 --2- 172.21.15.77:0/2578869152 >> v2:172.21.15.159:3300/0 conn(0x7efe000d03a0 0x7efe0015f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe0657d700 1 -- 172.21.15.77:0/2578869152 >> v1:172.21.15.159:6789/0 conn(0x7efe0015f700 legacy=0x7efe0009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.456+0000 7efe0657d700 1 -- 172.21.15.77:0/2578869152 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe000b2800 con 0x7efe000cffc0 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe0657d700 1 --2- 172.21.15.77:0/2578869152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe000b3320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009840 tx=0x7efdf4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7440e431cf894bf2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe0557b700 1 -- 172.21.15.77:0/2578869152 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4011830 con 0x7efe000cffc0 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe0557b700 1 -- 172.21.15.77:0/2578869152 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf4011990 con 0x7efe000cffc0 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe0557b700 1 -- 172.21.15.77:0/2578869152 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4011d20 con 0x7efe000cffc0 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 -- 172.21.15.77:0/2578869152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 msgr2=0x7efe000b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 --2- 172.21.15.77:0/2578869152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe000b3320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009840 tx=0x7efdf4009b50 comp rx=0 tx=0).stop 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 -- 172.21.15.77:0/2578869152 shutdown_connections 2024-01-15T14:36:10.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 --2- 172.21.15.77:0/2578869152 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe000b3320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 --2- 172.21.15.77:0/2578869152 >> v2:172.21.15.159:3300/0 conn(0x7efe000d03a0 0x7efe0015f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 -- 172.21.15.77:0/2578869152 >> 172.21.15.77:0/2578869152 conn(0x7efe000d1e50 msgr2=0x7efe000d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:10.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 -- 172.21.15.77:0/2578869152 shutdown_connections 2024-01-15T14:36:10.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.457+0000 7efe087e1700 1 -- 172.21.15.77:0/2578869152 wait complete. 2024-01-15T14:36:10.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 Processor -- start 2024-01-15T14:36:10.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 -- start start 2024-01-15T14:36:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe00154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 0x7efe000cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe0015f700 0x7efe000cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe000c9d20 con 0x7efe000d03a0 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe000c90a0 con 0x7efe0015f700 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe087e1700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe000c9380 con 0x7efe000cffc0 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe05d7c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 0x7efe000cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe05d7c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 0x7efe000cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37088/0 (socket says 172.21.15.77:37088) 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.459+0000 7efe05d7c700 1 -- 172.21.15.77:0/4004182255 learned_addr learned my addr 172.21.15.77:0/4004182255 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe0657d700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe00154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe06d7e700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe0015f700 0x7efe000cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe05d7c700 1 -- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 msgr2=0x7efe00154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe05d7c700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe00154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe05d7c700 1 -- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe0015f700 msgr2=0x7efe000cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe05d7c700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe0015f700 0x7efe000cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe05d7c700 1 -- 172.21.15.77:0/4004182255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf40094a0 con 0x7efe000d03a0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe05d7c700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 0x7efe000cf100 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00e9b0 tx=0x7efdfc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc009710 con 0x7efe000d03a0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc00ee50 con 0x7efe000d03a0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc00ca50 con 0x7efe000d03a0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe000c9660 con 0x7efe000d03a0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.460+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe001548c0 con 0x7efe000d03a0 2024-01-15T14:36:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.461+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe000c6500 con 0x7efe000d03a0 2024-01-15T14:36:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.462+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efdfc00cbb0 con 0x7efe000d03a0 2024-01-15T14:36:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.462+0000 7efdf2ffd700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efde406b350 0x7efde406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.462+0000 7efe0657d700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efde406b350 0x7efde406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.462+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdfc089ac0 con 0x7efe000d03a0 2024-01-15T14:36:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.463+0000 7efe0657d700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efde406b350 0x7efde406d810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000c00 tx=0x7efdf401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.467+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdfc0549c0 con 0x7efe000d03a0 2024-01-15T14:36:10.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.689+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7efe000cc780 con 0x7efde406b350 2024-01-15T14:36:10.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.694+0000 7efdf2ffd700 1 -- 172.21.15.77:0/4004182255 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7efdf4011b80 con 0x7efde406b350 2024-01-15T14:36:10.695 INFO:teuthology.orchestra.run.smithi077.stdout:/volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82 2024-01-15T14:36:10.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efde406b350 msgr2=0x7efde406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efde406b350 0x7efde406d810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000c00 tx=0x7efdf401c040 comp rx=0 tx=0).stop 2024-01-15T14:36:10.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 msgr2=0x7efe000cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:10.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 0x7efe000cf100 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00e9b0 tx=0x7efdfc00ecc0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 shutdown_connections 2024-01-15T14:36:10.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7efde406b350 0x7efde406d810 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7efe000cffc0 0x7efe00154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7efe0015f700 0x7efe000cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 --2- 172.21.15.77:0/4004182255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe000d03a0 0x7efe000cf100 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 >> 172.21.15.77:0/4004182255 conn(0x7efe000d1e50 msgr2=0x7efe000ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 shutdown_connections 2024-01-15T14:36:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:10.696+0000 7efe087e1700 1 -- 172.21.15.77:0/4004182255 wait complete. 2024-01-15T14:36:10.710 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82 2024-01-15T14:36:10.710 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:36:10.710 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:10.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:10.729 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T14:36:16.362 INFO:teuthology.orchestra.run.smithi077.stderr:1024+0 records in 2024-01-15T14:36:16.362 INFO:teuthology.orchestra.run.smithi077.stderr:1024+0 records out 2024-01-15T14:36:16.362 INFO:teuthology.orchestra.run.smithi077.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.56615 s, 193 MB/s 2024-01-15T14:36:16.363 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:36:16.363 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:16.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:16.382 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T14:36:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:1024+0 records in 2024-01-15T14:36:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:1024+0 records out 2024-01-15T14:36:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.70033 s, 188 MB/s 2024-01-15T14:36:22.141 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:36:22.141 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:22.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:22.158 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T14:36:27.782 INFO:teuthology.orchestra.run.smithi077.stderr:1024+0 records in 2024-01-15T14:36:27.783 INFO:teuthology.orchestra.run.smithi077.stderr:1024+0 records out 2024-01-15T14:36:27.783 INFO:teuthology.orchestra.run.smithi077.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.56534 s, 193 MB/s 2024-01-15T14:36:27.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T14:36:27.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.910+0000 7f96c1867700 1 Processor -- start 2024-01-15T14:36:27.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.910+0000 7f96c1867700 1 -- start start 2024-01-15T14:36:27.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96c1867700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f96bc161cb0 0x7f96bc0b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96c1867700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc0b41a0 con 0x7f96bc06c000 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96c1867700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc06f980 con 0x7f96bc06e0a0 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96c1867700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96bc162290 con 0x7f96bc161cb0 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96baffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f96bc06c000 0x7f96bc06c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:37656/0 (socket says 172.21.15.77:37656) 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96baffd700 1 -- 172.21.15.77:0/1330764302 learned_addr learned my addr 172.21.15.77:0/1330764302 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055978569 0 0) 0x7f96bc0b41a0 con 0x7f96bc06c000 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a40035b0 con 0x7f96bc06c000 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.911+0000 7f96ba7fc700 1 --2- 172.21.15.77:0/1330764302 >> v2:172.21.15.159:3300/0 conn(0x7f96bc161cb0 0x7f96bc0b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2668632148 0 0) 0x7f96a40035b0 con 0x7f96bc06c000 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 >> v2:172.21.15.159:3300/0 conn(0x7f96bc161cb0 msgr2=0x7f96bc0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:27.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 --2- 172.21.15.77:0/1330764302 >> v2:172.21.15.159:3300/0 conn(0x7f96bc161cb0 0x7f96bc0b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 >> v1:172.21.15.159:6789/0 conn(0x7f96bc06e0a0 legacy=0x7f96bc06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc162570 con 0x7f96bc06c000 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f96b0001670 con 0x7f96bc06c000 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 >> v1:172.21.15.77:6789/0 conn(0x7f96bc06c000 legacy=0x7f96bc06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96a4003a00 0x7f96a4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96a4004850 0x7f96a4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96a400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc162290 con 0x7f96a4005210 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a400c0b0 con 0x7f96a4003a00 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a400c270 con 0x7f96a4004850 2024-01-15T14:36:27.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96a400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96baffd700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96a4003a00 0x7f96a4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96ba7fc700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96a4004850 0x7f96a4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 -- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96a4004850 msgr2=0x7f96a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96a4004850 0x7f96a4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 -- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96a4003a00 msgr2=0x7f96a4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96a4003a00 0x7f96a4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 -- 172.21.15.77:0/1330764302 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a40035b0 con 0x7f96a4005210 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 -- 172.21.15.77:0/1330764302 reap_dead start 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96bb7fe700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96a400ba10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f96b400c190 tx=0x7f96a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172047b1b40528d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4001db0 con 0x7f96a4005210 2024-01-15T14:36:27.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b400ccf0 con 0x7f96a4005210 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.912+0000 7f96b9ffb700 1 -- 172.21.15.77:0/1330764302 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4003c40 con 0x7f96a4005210 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 -- 172.21.15.77:0/1330764302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 msgr2=0x7f96a400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96a400ba10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f96b400c190 tx=0x7f96a4005880 comp rx=0 tx=0).stop 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 -- 172.21.15.77:0/1330764302 shutdown_connections 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 --2- 172.21.15.77:0/1330764302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96a400ba10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 -- 172.21.15.77:0/1330764302 >> 172.21.15.77:0/1330764302 conn(0x7f96bc071bb0 msgr2=0x7f96bc071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 -- 172.21.15.77:0/1330764302 shutdown_connections 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.913+0000 7f96c1867700 1 -- 172.21.15.77:0/1330764302 wait complete. 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.914+0000 7f96c1867700 1 Processor -- start 2024-01-15T14:36:27.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.914+0000 7f96c1867700 1 -- start start 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96c1867700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96bc1f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96c1867700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96bc160880 0x7f96bc0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96c1867700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96bc15e5d0 0x7f96bc1f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96c1867700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1f4bc0 con 0x7f96a4005210 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96c1867700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1f4320 con 0x7f96bc15e5d0 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96c1867700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1f4600 con 0x7f96bc160880 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96bc1f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96bc1f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47520/0 (socket says 172.21.15.77:47520) 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 -- 172.21.15.77:0/1786201952 learned_addr learned my addr 172.21.15.77:0/1786201952 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96ba7fc700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96bc160880 0x7f96bc0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96bb7fe700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96bc15e5d0 0x7f96bc1f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 -- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96bc160880 msgr2=0x7f96bc0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96bc160880 0x7f96bc0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 -- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96bc15e5d0 msgr2=0x7f96bc1f77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96bc15e5d0 0x7f96bc1f77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.915+0000 7f96baffd700 1 -- 172.21.15.77:0/1786201952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc1f48e0 con 0x7f96a4005210 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.916+0000 7f96baffd700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96bc1f3c10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f96b00072c0 tx=0x7f96b00075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.916+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b000f6c0 con 0x7f96a4005210 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.916+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b000f840 con 0x7f96a4005210 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.916+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b000fbf0 con 0x7f96a4005210 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.916+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc1f72d0 con 0x7f96a4005210 2024-01-15T14:36:27.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.916+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc160d00 con 0x7f96a4005210 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.917+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc1f74f0 con 0x7f96a4005210 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.917+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f96b000fd50 con 0x7f96a4005210 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.918+0000 7f96a3fff700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f96a806b3e0 0x7f96a806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.918+0000 7f96ba7fc700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f96a806b3e0 0x7f96a806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.918+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96b0089aa0 con 0x7f96a4005210 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.918+0000 7f96ba7fc700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f96a806b3e0 0x7f96a806d8a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f96bc1447c0 tx=0x7f96ac009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:27.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:27.922+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96b0054920 con 0x7f96a4005210 2024-01-15T14:36:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.116+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f96bc069020 con 0x7f96a4005210 2024-01-15T14:36:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.119+0000 7f96a3fff700 1 -- 172.21.15.77:0/1786201952 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f96b0057f40 con 0x7f96a4005210 2024-01-15T14:36:28.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f96a806b3e0 msgr2=0x7f96a806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f96a806b3e0 0x7f96a806d8a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f96bc1447c0 tx=0x7f96ac009210 comp rx=0 tx=0).stop 2024-01-15T14:36:28.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 msgr2=0x7f96bc1f3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96bc1f3c10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f96b00072c0 tx=0x7f96b00075d0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 shutdown_connections 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f96a806b3e0 0x7f96a806d8a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f96bc15e5d0 0x7f96bc1f77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96a4005210 0x7f96bc1f3c10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 --2- 172.21.15.77:0/1786201952 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f96bc160880 0x7f96bc0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 >> 172.21.15.77:0/1786201952 conn(0x7f96bc071bb0 msgr2=0x7f96bc071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 shutdown_connections 2024-01-15T14:36:28.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.121+0000 7f96c1867700 1 -- 172.21.15.77:0/1786201952 wait complete. 2024-01-15T14:36:28.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T14:36:28.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.262+0000 7f4e24843700 1 Processor -- start 2024-01-15T14:36:28.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.262+0000 7f4e24843700 1 -- start start 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e24843700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e200b01e0 0x7f4e2006e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e24843700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200746a0 0x7f4e200b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e24843700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200b05c0 con 0x7f4e200b59c0 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e24843700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e201470a0 con 0x7f4e200b01e0 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e24843700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e200b5e70 con 0x7f4e200746a0 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e200b01e0 0x7f4e2006e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1dd9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4e200b59c0 0x7f4e20073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:37668/0 (socket says 172.21.15.77:37668) 2024-01-15T14:36:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1dd9b700 1 -- 172.21.15.77:0/4180873341 learned_addr learned my addr 172.21.15.77:0/4180873341 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1ed9d700 1 --2- 172.21.15.77:0/4180873341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200746a0 0x7f4e200b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1e59c700 1 -- 172.21.15.77:0/4180873341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200746a0 msgr2=0x7f4e200b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1e59c700 1 --2- 172.21.15.77:0/4180873341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200746a0 0x7f4e200b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1e59c700 1 -- 172.21.15.77:0/4180873341 >> v1:172.21.15.77:6789/0 conn(0x7f4e200b59c0 legacy=0x7f4e20073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1e59c700 1 -- 172.21.15.77:0/4180873341 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200b4250 con 0x7f4e200b01e0 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1d59a700 1 -- 172.21.15.77:0/4180873341 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810433318 0 0) 0x7f4e200b05c0 con 0x7f4e200b59c0 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.263+0000 7f4e1e59c700 1 --2- 172.21.15.77:0/4180873341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e2006e9c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009870 tx=0x7f4e10009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76d9360181a89af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e1d59a700 1 -- 172.21.15.77:0/4180873341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10011830 con 0x7f4e200b01e0 2024-01-15T14:36:28.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e1d59a700 1 -- 172.21.15.77:0/4180873341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e10011990 con 0x7f4e200b01e0 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e1d59a700 1 -- 172.21.15.77:0/4180873341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10011d20 con 0x7f4e200b01e0 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 -- 172.21.15.77:0/4180873341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 msgr2=0x7f4e2006e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 --2- 172.21.15.77:0/4180873341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e2006e9c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009870 tx=0x7f4e10009b80 comp rx=0 tx=0).stop 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 -- 172.21.15.77:0/4180873341 shutdown_connections 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 --2- 172.21.15.77:0/4180873341 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200746a0 0x7f4e200b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 --2- 172.21.15.77:0/4180873341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e2006e9c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 -- 172.21.15.77:0/4180873341 >> 172.21.15.77:0/4180873341 conn(0x7f4e200d0b40 msgr2=0x7f4e200d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 -- 172.21.15.77:0/4180873341 shutdown_connections 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.264+0000 7f4e24843700 1 -- 172.21.15.77:0/4180873341 wait complete. 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 Processor -- start 2024-01-15T14:36:28.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 -- start start 2024-01-15T14:36:28.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e200746a0 0x7f4e2013f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e201425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200b59c0 0x7f4e200aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2013ee60 con 0x7f4e200b01e0 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e200af200 con 0x7f4e200746a0 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.266+0000 7f4e24843700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2013f0d0 con 0x7f4e200b59c0 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e201425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e201425f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47548/0 (socket says 172.21.15.77:47548) 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 -- 172.21.15.77:0/4001293681 learned_addr learned my addr 172.21.15.77:0/4001293681 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1ed9d700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200b59c0 0x7f4e200aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1e59c700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e200746a0 0x7f4e2013f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 -- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200b59c0 msgr2=0x7f4e200aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200b59c0 0x7f4e200aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 -- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e200746a0 msgr2=0x7f4e2013f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e200746a0 0x7f4e2013f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 -- 172.21.15.77:0/4001293681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100094a0 con 0x7f4e200b01e0 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e1dd9b700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e201425f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400c550 tx=0x7f4e1400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14004c80 con 0x7f4e200b01e0 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e14004e00 con 0x7f4e200b01e0 2024-01-15T14:36:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e20141b10 con 0x7f4e200b01e0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14007310 con 0x7f4e200b01e0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.267+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e201420d0 con 0x7f4e200b01e0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.268+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e201422f0 con 0x7f4e200b01e0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.269+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f4e14007470 con 0x7f4e200b01e0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.269+0000 7f4e0effd700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4e0006b3e0 0x7f4e0006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.269+0000 7f4e1e59c700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4e0006b3e0 0x7f4e0006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.269+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e14089760 con 0x7f4e200b01e0 2024-01-15T14:36:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.270+0000 7f4e1e59c700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4e0006b3e0 0x7f4e0006d8a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e100097c0 tx=0x7f4e10008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:28.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.274+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e140545e0 con 0x7f4e200b01e0 2024-01-15T14:36:28.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.502+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e20146ee0 con 0x7f4e0006b3e0 2024-01-15T14:36:28.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.515+0000 7f4e0effd700 1 -- 172.21.15.77:0/4001293681 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4e20146ee0 con 0x7f4e0006b3e0 2024-01-15T14:36:28.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4e0006b3e0 msgr2=0x7f4e0006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4e0006b3e0 0x7f4e0006d8a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e100097c0 tx=0x7f4e10008030 comp rx=0 tx=0).stop 2024-01-15T14:36:28.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 msgr2=0x7f4e201425f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e201425f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400c550 tx=0x7f4e1400c860 comp rx=0 tx=0).stop 2024-01-15T14:36:28.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 shutdown_connections 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f4e0006b3e0 0x7f4e0006d8a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4e200b59c0 0x7f4e200aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f4e200746a0 0x7f4e2013f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 --2- 172.21.15.77:0/4001293681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200b01e0 0x7f4e201425f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.517+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 >> 172.21.15.77:0/4001293681 conn(0x7f4e200d0b40 msgr2=0x7f4e20072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.518+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 shutdown_connections 2024-01-15T14:36:28.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.518+0000 7f4e24843700 1 -- 172.21.15.77:0/4001293681 wait complete. 2024-01-15T14:36:28.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T14:36:28.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.655+0000 7f8b3079c700 1 Processor -- start 2024-01-15T14:36:28.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.655+0000 7f8b3079c700 1 -- start start 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b3079c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8b28145790 0x7f8b280b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b3079c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f8b280746a0 0x7f8b280b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b3079c700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2806c100 con 0x7f8b2806aff0 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b3079c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b280ae0f0 con 0x7f8b28145790 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b3079c700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b28145b70 con 0x7f8b280746a0 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8b28145790 0x7f8b280b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8b28145790 0x7f8b280b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47550/0 (socket says 172.21.15.77:47550) 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 -- 172.21.15.77:0/4173649290 learned_addr learned my addr 172.21.15.77:0/4173649290 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2ed39700 1 --2- 172.21.15.77:0/4173649290 >> v2:172.21.15.159:3300/0 conn(0x7f8b280746a0 0x7f8b280b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 -- 172.21.15.77:0/4173649290 >> v2:172.21.15.159:3300/0 conn(0x7f8b280746a0 msgr2=0x7f8b280b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 --2- 172.21.15.77:0/4173649290 >> v2:172.21.15.159:3300/0 conn(0x7f8b280746a0 0x7f8b280b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 -- 172.21.15.77:0/4173649290 >> v1:172.21.15.159:6789/0 conn(0x7f8b2806aff0 legacy=0x7f8b28073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.656+0000 7f8b2e538700 1 -- 172.21.15.77:0/4173649290 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b280b0660 con 0x7f8b28145790 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b2e538700 1 --2- 172.21.15.77:0/4173649290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b0120 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8b280d1c80 tx=0x7f8b18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e540639a3a4e01 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b2d536700 1 -- 172.21.15.77:0/4173649290 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800b860 con 0x7f8b28145790 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b2d536700 1 -- 172.21.15.77:0/4173649290 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1800b9c0 con 0x7f8b28145790 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b2d536700 1 -- 172.21.15.77:0/4173649290 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800bd50 con 0x7f8b28145790 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 -- 172.21.15.77:0/4173649290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 msgr2=0x7f8b280b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 --2- 172.21.15.77:0/4173649290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b0120 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8b280d1c80 tx=0x7f8b18009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 -- 172.21.15.77:0/4173649290 shutdown_connections 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 --2- 172.21.15.77:0/4173649290 >> v2:172.21.15.159:3300/0 conn(0x7f8b280746a0 0x7f8b280b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 --2- 172.21.15.77:0/4173649290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b0120 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 -- 172.21.15.77:0/4173649290 >> 172.21.15.77:0/4173649290 conn(0x7f8b280b3250 msgr2=0x7f8b280d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 -- 172.21.15.77:0/4173649290 shutdown_connections 2024-01-15T14:36:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.657+0000 7f8b3079c700 1 -- 172.21.15.77:0/4173649290 wait complete. 2024-01-15T14:36:28.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 Processor -- start 2024-01-15T14:36:28.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 -- start start 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b2806aff0 0x7f8b2815b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b280746a0 0x7f8b280b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b280cca20 con 0x7f8b28145790 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2815da20 con 0x7f8b280746a0 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b3079c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b280cc460 con 0x7f8b2806aff0 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b2ed39700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.659+0000 7f8b2ed39700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47566/0 (socket says 172.21.15.77:47566) 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 -- 172.21.15.77:0/3215592875 learned_addr learned my addr 172.21.15.77:0/3215592875 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2dd37700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b280746a0 0x7f8b280b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 -- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b2806aff0 msgr2=0x7f8b2815b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2e538700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b2806aff0 0x7f8b2815b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b2806aff0 0x7f8b2815b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 -- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b280746a0 msgr2=0x7f8b280b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b280746a0 0x7f8b280b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 -- 172.21.15.77:0/3215592875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18009550 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2dd37700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b280746a0 0x7f8b280b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2e538700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b2806aff0 0x7f8b2815b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b2ed39700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b4670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8b20007de0 tx=0x7f8b2000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000ede0 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b20014420 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b200147d0 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b280cc740 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.660+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b280cb6f0 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.661+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b28069020 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.661+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f8b20014930 con 0x7f8b28145790 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.662+0000 7f8b1effd700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8b0c06b3e0 0x7f8b0c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.662+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b2008a330 con 0x7f8b28145790 2024-01-15T14:36:28.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.662+0000 7f8b2e538700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8b0c06b3e0 0x7f8b0c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:28.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.662+0000 7f8b2e538700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8b0c06b3e0 0x7f8b0c06d8a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b280affd0 tx=0x7f8b18008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:28.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.666+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b200551b0 con 0x7f8b28145790 2024-01-15T14:36:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.891+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b2806bab0 con 0x7f8b0c06b3e0 2024-01-15T14:36:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.896+0000 7f8b1effd700 1 -- 172.21.15.77:0/3215592875 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f8b1801f070 con 0x7f8b0c06b3e0 2024-01-15T14:36:28.897 INFO:teuthology.orchestra.run.smithi077.stdout:/volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82 2024-01-15T14:36:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8b0c06b3e0 msgr2=0x7f8b0c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8b0c06b3e0 0x7f8b0c06d8a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b280affd0 tx=0x7f8b18008030 comp rx=0 tx=0).stop 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 msgr2=0x7f8b280b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b4670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8b20007de0 tx=0x7f8b2000e3c0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 shutdown_connections 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f8b0c06b3e0 0x7f8b0c06d8a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f8b2806aff0 0x7f8b2815b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b28145790 0x7f8b280b4670 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 --2- 172.21.15.77:0/3215592875 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f8b280746a0 0x7f8b280b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 >> 172.21.15.77:0/3215592875 conn(0x7f8b280b3250 msgr2=0x7f8b281432b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 shutdown_connections 2024-01-15T14:36:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:28.898+0000 7f8b3079c700 1 -- 172.21.15.77:0/3215592875 wait complete. 2024-01-15T14:36:28.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:28.911 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82) 2024-01-15T14:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout:# file: volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82 2024-01-15T14:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph.dir.rbytes="2972172288" 2024-01-15T14:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:28.929 INFO:tasks.cephfs.test_volumes:r_size = 2972172288 2024-01-15T14:36:30.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:30.930 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82) 2024-01-15T14:36:30.952 INFO:teuthology.orchestra.run.smithi077.stdout:# file: volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82 2024-01-15T14:36:30.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph.dir.rbytes="2972172288" 2024-01-15T14:36:30.952 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:30.952 INFO:tasks.cephfs.test_volumes:r_size = 2972172288 2024-01-15T14:36:32.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:32.953 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82) 2024-01-15T14:36:32.977 INFO:teuthology.orchestra.run.smithi077.stdout:# file: volumes/_nogroup/sv1/799270c2-582e-4db6-a621-f67a45a41d82 2024-01-15T14:36:32.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T14:36:32.977 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:32.977 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T14:36:32.978 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T14:36:33.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.156+0000 7fd379379700 1 Processor -- start 2024-01-15T14:36:33.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.156+0000 7fd379379700 1 -- start start 2024-01-15T14:36:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd379379700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fd3740d0080 0x7fd37406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd379379700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 0x7fd3740af520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd379379700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374070e90 con 0x7fd3740d0460 2024-01-15T14:36:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd379379700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3740d1750 con 0x7fd3740d0080 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd379379700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740705f0 con 0x7fd3740af0b0 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd3727fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd3740d0460 0x7fd3740d2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45212/0 (socket says 172.21.15.77:45212) 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd3727fc700 1 -- 172.21.15.77:0/3209147334 learned_addr learned my addr 172.21.15.77:0/3209147334 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968707160 0 0) 0x7fd374070e90 con 0x7fd3740d0460 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd3737fe700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 0x7fd3740af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd372ffd700 1 --2- 172.21.15.77:0/3209147334 >> v2:172.21.15.159:3300/0 conn(0x7fd3740d0080 0x7fd37406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.157+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3600035b0 con 0x7fd3740d0460 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226303052 0 0) 0x7fd3600035b0 con 0x7fd3740d0460 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 msgr2=0x7fd3740af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 0x7fd3740af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 >> v2:172.21.15.159:3300/0 conn(0x7fd3740d0080 msgr2=0x7fd37406ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 --2- 172.21.15.77:0/3209147334 >> v2:172.21.15.159:3300/0 conn(0x7fd3740d0080 0x7fd37406ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740708d0 con 0x7fd3740d0460 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fd35c001670 con 0x7fd3740d0460 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 >> v1:172.21.15.77:6789/0 conn(0x7fd3740d0460 legacy=0x7fd3740d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd360003a00 0x7fd360003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 0x7fd360004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd360004d90 0x7fd360005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740705f0 con 0x7fd360004d90 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36000ba10 con 0x7fd360003a00 2024-01-15T14:36:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36000bb50 con 0x7fd3740af0b0 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd372ffd700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd360003a00 0x7fd360003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd360004d90 0x7fd360005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3727fc700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 0x7fd360004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 -- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 msgr2=0x7fd360004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740af0b0 0x7fd360004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 -- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd360003a00 msgr2=0x7fd360003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd360003a00 0x7fd360003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 -- 172.21.15.77:0/3209147334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3600035b0 con 0x7fd360004d90 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd3737fe700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd360004d90 0x7fd360005210 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009fc0 tx=0x7fd360005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89eea595fdb9f863 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c00ec30 con 0x7fd360004d90 2024-01-15T14:36:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c00f040 con 0x7fd360004d90 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.158+0000 7fd371ffb700 1 -- 172.21.15.77:0/3209147334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c014480 con 0x7fd360004d90 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd379379700 1 -- 172.21.15.77:0/3209147334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd360004d90 msgr2=0x7fd360005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd379379700 1 --2- 172.21.15.77:0/3209147334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd360004d90 0x7fd360005210 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009fc0 tx=0x7fd360005880 comp rx=0 tx=0).stop 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd3737fe700 1 -- 172.21.15.77:0/3209147334 reap_dead start 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd379379700 1 -- 172.21.15.77:0/3209147334 shutdown_connections 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd379379700 1 -- 172.21.15.77:0/3209147334 >> 172.21.15.77:0/3209147334 conn(0x7fd37409aa00 msgr2=0x7fd37409ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd379379700 1 -- 172.21.15.77:0/3209147334 shutdown_connections 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.159+0000 7fd379379700 1 -- 172.21.15.77:0/3209147334 wait complete. 2024-01-15T14:36:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.160+0000 7fd379379700 1 Processor -- start 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 -- start start 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740c7de0 0x7fd3740cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 0x7fd3740b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740c81c0 0x7fd3740b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740ca250 con 0x7fd3740c9d50 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740ca530 con 0x7fd3740c81c0 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd379379700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740ca810 con 0x7fd3740c7de0 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd3727fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 0x7fd3740b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd3727fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 0x7fd3740b7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38358/0 (socket says 172.21.15.77:38358) 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd3727fc700 1 -- 172.21.15.77:0/24880627 learned_addr learned my addr 172.21.15.77:0/24880627 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd372ffd700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740c7de0 0x7fd3740cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.161+0000 7fd3737fe700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740c81c0 0x7fd3740b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3727fc700 1 -- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740c7de0 msgr2=0x7fd3740cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3727fc700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740c7de0 0x7fd3740cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3727fc700 1 -- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740c81c0 msgr2=0x7fd3740b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3727fc700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740c81c0 0x7fd3740b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3727fc700 1 -- 172.21.15.77:0/24880627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740caaf0 con 0x7fd3740c9d50 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3737fe700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740c81c0 0x7fd3740b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd3727fc700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 0x7fd3740b7360 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd35c000fb0 tx=0x7fd35c0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00f8a0 con 0x7fd3740c9d50 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c00fa00 con 0x7fd3740c9d50 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3740cadd0 con 0x7fd3740c9d50 2024-01-15T14:36:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00fdb0 con 0x7fd3740c9d50 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.162+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3740b7a80 con 0x7fd3740c9d50 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.163+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3740caff0 con 0x7fd3740c9d50 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.163+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fd35c01d460 con 0x7fd3740c9d50 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.164+0000 7fd36b7fe700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35406b3e0 0x7fd35406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.164+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd35c089c40 con 0x7fd3740c9d50 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.164+0000 7fd372ffd700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35406b3e0 0x7fd35406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.164+0000 7fd372ffd700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35406b3e0 0x7fd35406d8a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ba0 tx=0x7fd364008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:33.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.168+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd35c054b50 con 0x7fd3740c9d50 2024-01-15T14:36:33.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:36:33.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:36:33.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.406+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3740d0910 con 0x7fd35406b3e0 2024-01-15T14:36:33.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.463+0000 7fd36b7fe700 1 -- 172.21.15.77:0/24880627 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd3640097f0 con 0x7fd35406b3e0 2024-01-15T14:36:33.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.465+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35406b3e0 msgr2=0x7fd35406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.465+0000 7fd379379700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35406b3e0 0x7fd35406d8a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ba0 tx=0x7fd364008040 comp rx=0 tx=0).stop 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.465+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 msgr2=0x7fd3740b7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.465+0000 7fd379379700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 0x7fd3740b7360 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd35c000fb0 tx=0x7fd35c0076a0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 shutdown_connections 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fd35406b3e0 0x7fd35406d8a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fd3740c7de0 0x7fd3740cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fd3740c81c0 0x7fd3740b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 --2- 172.21.15.77:0/24880627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3740c9d50 0x7fd3740b7360 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 >> 172.21.15.77:0/24880627 conn(0x7fd37409aa00 msgr2=0x7fd3740b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 shutdown_connections 2024-01-15T14:36:33.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.466+0000 7fd379379700 1 -- 172.21.15.77:0/24880627 wait complete. 2024-01-15T14:36:33.468 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.467+0000 7feba3229700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.468 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.467+0000 7feba3229700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.468 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.467+0000 7feba3229700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.468 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.467+0000 7feba3229700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.468 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.467+0000 7feba3229700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.480 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T14:36:33.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:33.482 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.481+0000 7feba2a28700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.482 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.481+0000 7feba2a28700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.482 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.481+0000 7feba2a28700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.482 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.481+0000 7feba2a28700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.482 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:33.481+0000 7feba2a28700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.612+0000 7f421e2d0700 1 Processor -- start 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.612+0000 7f421e2d0700 1 -- start start 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421e2d0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4218161690 0x7f42180d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421e2d0700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f4218161a70 0x7f42180d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421e2d0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42180d1730 0x7f421809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421e2d0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42180d0a70 con 0x7f4218161690 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421e2d0700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42180d1c10 con 0x7f4218161a70 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421e2d0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180b2ab0 con 0x7f42180d1730 2024-01-15T14:36:33.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4218161690 0x7f42180d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4218161690 0x7f42180d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38404/0 (socket says 172.21.15.77:38404) 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 -- 172.21.15.77:0/2856908751 learned_addr learned my addr 172.21.15.77:0/2856908751 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f42177fe700 1 --2- 172.21.15.77:0/2856908751 >> v2:172.21.15.159:3300/0 conn(0x7f4218161a70 0x7f42180d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 -- 172.21.15.77:0/2856908751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42180d1730 msgr2=0x7f421809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421c86d700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42180d1730 0x7f421809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42180d1730 0x7f421809aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 -- 172.21.15.77:0/2856908751 >> v2:172.21.15.159:3300/0 conn(0x7f4218161a70 msgr2=0x7f42180d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 --2- 172.21.15.77:0/2856908751 >> v2:172.21.15.159:3300/0 conn(0x7f4218161a70 0x7f42180d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 -- 172.21.15.77:0/2856908751 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180b2d90 con 0x7f4218161690 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f42177fe700 1 --2- 172.21.15.77:0/2856908751 >> v2:172.21.15.159:3300/0 conn(0x7f4218161a70 0x7f42180d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f421c86d700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42180d1730 0x7f421809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.613+0000 7f4217fff700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180d0f20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4218072ec0 tx=0x7f420c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aac07013d512e8a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f4216ffd700 1 -- 172.21.15.77:0/2856908751 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c00b870 con 0x7f4218161690 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f4216ffd700 1 -- 172.21.15.77:0/2856908751 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c00b9d0 con 0x7f4218161690 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f4216ffd700 1 -- 172.21.15.77:0/2856908751 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c00bd60 con 0x7f4218161690 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 -- 172.21.15.77:0/2856908751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 msgr2=0x7f42180d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180d0f20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4218072ec0 tx=0x7f420c009bd0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 -- 172.21.15.77:0/2856908751 shutdown_connections 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f42180d1730 0x7f421809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2856908751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180d0f20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2856908751 >> v2:172.21.15.159:3300/0 conn(0x7f4218161a70 0x7f42180d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 -- 172.21.15.77:0/2856908751 >> 172.21.15.77:0/2856908751 conn(0x7f42180d1e50 msgr2=0x7f42180d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:33.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 -- 172.21.15.77:0/2856908751 shutdown_connections 2024-01-15T14:36:33.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.614+0000 7f421e2d0700 1 -- 172.21.15.77:0/2856908751 wait complete. 2024-01-15T14:36:33.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.615+0000 7f421e2d0700 1 Processor -- start 2024-01-15T14:36:33.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 -- start start 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42180d1730 0x7f42180cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4218161a70 0x7f42180cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180af0c0 con 0x7f4218161690 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180af3a0 con 0x7f42180d1730 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f421e2d0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180af680 con 0x7f4218161a70 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f42177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.616+0000 7f42177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180b0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38416/0 (socket says 172.21.15.77:38416) 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 -- 172.21.15.77:0/2599784782 learned_addr learned my addr 172.21.15.77:0/2599784782 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f4217fff700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42180d1730 0x7f42180cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 -- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4218161a70 msgr2=0x7f42180cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f421c86d700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4218161a70 0x7f42180cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4218161a70 0x7f42180cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 -- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42180d1730 msgr2=0x7f42180cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42180d1730 0x7f42180cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 -- 172.21.15.77:0/2599784782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c009550 con 0x7f4218161690 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f4217fff700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42180d1730 0x7f42180cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f421c86d700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4218161a70 0x7f42180cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f42177fe700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180b0160 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f420800c7f0 tx=0x7f420800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208011800 con 0x7f4218161690 2024-01-15T14:36:33.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4208011960 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42180af960 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208011d10 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.617+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42180aece0 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.618+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42180afe80 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.619+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f4208018360 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.619+0000 7f4214ff9700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f420406b3e0 0x7f420406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.619+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4208089b70 con 0x7f4218161690 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.619+0000 7f4217fff700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f420406b3e0 0x7f420406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:33.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.619+0000 7f4217fff700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f420406b3e0 0x7f420406d8a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f420c009130 tx=0x7f420c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:33.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.623+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42180afe80 con 0x7f4218161690 2024-01-15T14:36:33.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:33.860+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f421809af40 con 0x7f420406b3e0 2024-01-15T14:36:34.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:34.277+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f4208054d40 con 0x7f4218161690 2024-01-15T14:36:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.754+0000 7f4214ff9700 1 -- 172.21.15.77:0/2599784782 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f420c00b670 con 0x7f420406b3e0 2024-01-15T14:36:36.754 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:36.754 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:36.755 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:36.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f420406b3e0 msgr2=0x7f420406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:36.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f420406b3e0 0x7f420406d8a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f420c009130 tx=0x7f420c008030 comp rx=0 tx=0).stop 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 msgr2=0x7f42180b0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180b0160 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f420800c7f0 tx=0x7f420800cbb0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 shutdown_connections 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f420406b3e0 0x7f420406d8a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f4218161a70 0x7f42180cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f42180d1730 0x7f42180cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 --2- 172.21.15.77:0/2599784782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4218161690 0x7f42180b0160 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 >> 172.21.15.77:0/2599784782 conn(0x7f42180d1e50 msgr2=0x7f42180cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 shutdown_connections 2024-01-15T14:36:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.757+0000 7f421e2d0700 1 -- 172.21.15.77:0/2599784782 wait complete. 2024-01-15T14:36:36.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T14:36:36.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.899+0000 7f5f39c7b700 1 Processor -- start 2024-01-15T14:36:36.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.899+0000 7f5f39c7b700 1 -- start start 2024-01-15T14:36:36.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f39c7b700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5f3415bc90 0x7f5f340d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f39c7b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f3415c070 0x7f5f340d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f39c7b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3406f060 0x7f5f3406f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f39c7b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3406fdd0 con 0x7f5f3415c070 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f39c7b700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340700b0 con 0x7f5f3415bc90 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f39c7b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34070390 con 0x7f5f3406f060 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f32ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f3415c070 0x7f5f340d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f32ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f3415c070 0x7f5f340d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38418/0 (socket says 172.21.15.77:38418) 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.900+0000 7f5f32ffd700 1 -- 172.21.15.77:0/4182798797 learned_addr learned my addr 172.21.15.77:0/4182798797 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f337fe700 1 --2- 172.21.15.77:0/4182798797 >> v2:172.21.15.159:3300/0 conn(0x7f5f3415bc90 0x7f5f340d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f33fff700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3406f060 0x7f5f3406f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f32ffd700 1 -- 172.21.15.77:0/4182798797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3406f060 msgr2=0x7f5f3406f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:36.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f32ffd700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3406f060 0x7f5f3406f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f32ffd700 1 -- 172.21.15.77:0/4182798797 >> v2:172.21.15.159:3300/0 conn(0x7f5f3415bc90 msgr2=0x7f5f340d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f32ffd700 1 --2- 172.21.15.77:0/4182798797 >> v2:172.21.15.159:3300/0 conn(0x7f5f3415bc90 0x7f5f340d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f32ffd700 1 -- 172.21.15.77:0/4182798797 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34070670 con 0x7f5f3415c070 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f337fe700 1 --2- 172.21.15.77:0/4182798797 >> v2:172.21.15.159:3300/0 conn(0x7f5f3415bc90 0x7f5f340d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f33fff700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3406f060 0x7f5f3406f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f32ffd700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415c070 0x7f5f340d06e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5f3409fcd0 tx=0x7f5f1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516ac8d6ada52b9d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f327fc700 1 -- 172.21.15.77:0/4182798797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00b870 con 0x7f5f3415c070 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f327fc700 1 -- 172.21.15.77:0/4182798797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c00b9d0 con 0x7f5f3415c070 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f327fc700 1 -- 172.21.15.77:0/4182798797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00bd60 con 0x7f5f3415c070 2024-01-15T14:36:36.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f39c7b700 1 -- 172.21.15.77:0/4182798797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415c070 msgr2=0x7f5f340d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415c070 0x7f5f340d06e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5f3409fcd0 tx=0x7f5f1c009bd0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.901+0000 7f5f39c7b700 1 -- 172.21.15.77:0/4182798797 shutdown_connections 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.902+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/4182798797 >> v2:172.21.15.159:3300/0 conn(0x7f5f3415bc90 0x7f5f340d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.902+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3406f060 0x7f5f3406f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.902+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/4182798797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415c070 0x7f5f340d06e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.902+0000 7f5f39c7b700 1 -- 172.21.15.77:0/4182798797 >> 172.21.15.77:0/4182798797 conn(0x7f5f34073130 msgr2=0x7f5f3409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.902+0000 7f5f39c7b700 1 -- 172.21.15.77:0/4182798797 shutdown_connections 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.902+0000 7f5f39c7b700 1 -- 172.21.15.77:0/4182798797 wait complete. 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.903+0000 7f5f39c7b700 1 Processor -- start 2024-01-15T14:36:36.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.903+0000 7f5f39c7b700 1 -- start start 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f39c7b700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f3406f060 0x7f5f3414bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f39c7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 0x7f5f340c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f39c7b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3415c070 0x7f5f3409ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f39c7b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3409ca40 con 0x7f5f3415bc90 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f39c7b700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3414c080 con 0x7f5f3406f060 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f39c7b700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3409be30 con 0x7f5f3415c070 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f32ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 0x7f5f340c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f32ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 0x7f5f340c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38426/0 (socket says 172.21.15.77:38426) 2024-01-15T14:36:36.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f32ffd700 1 -- 172.21.15.77:0/2470788276 learned_addr learned my addr 172.21.15.77:0/2470788276 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f337fe700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f3406f060 0x7f5f3414bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.904+0000 7f5f33fff700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3415c070 0x7f5f3409ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f32ffd700 1 -- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3415c070 msgr2=0x7f5f3409ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f32ffd700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3415c070 0x7f5f3409ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f32ffd700 1 -- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f3406f060 msgr2=0x7f5f3414bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f32ffd700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f3406f060 0x7f5f3414bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f32ffd700 1 -- 172.21.15.77:0/2470788276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c009550 con 0x7f5f3415bc90 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f32ffd700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 0x7f5f340c7680 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c014040 tx=0x7f5f1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c004310 con 0x7f5f3415bc90 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c01f070 con 0x7f5f3415bc90 2024-01-15T14:36:36.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0036a0 con 0x7f5f3415bc90 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3409c110 con 0x7f5f3415bc90 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.905+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3409c6d0 con 0x7f5f3415bc90 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.906+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3409cbf0 con 0x7f5f3415bc90 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.906+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f5f1c021030 con 0x7f5f3415bc90 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.907+0000 7f5f2bfff700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f1406b500 0x7f5f1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.907+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f1c08b080 con 0x7f5f3415bc90 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.907+0000 7f5f337fe700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f1406b500 0x7f5f1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:36.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.907+0000 7f5f337fe700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f1406b500 0x7f5f1406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400d540 tx=0x7f5f24007810 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:36.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:36.912+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f1c0bde90 con 0x7f5f3415bc90 2024-01-15T14:36:37.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.181+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5f34067f60 con 0x7f5f3415bc90 2024-01-15T14:36:37.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.181+0000 7f5f2bfff700 1 -- 172.21.15.77:0/2470788276 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+3262 (secure 0 0 0) 0x7f5f1c029090 con 0x7f5f3415bc90 2024-01-15T14:36:37.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "fsid": "a5eae108-559b-4366-a994-4d2898914163", 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "health": { 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "HEALTH_OK", 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "checks": {}, 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "mutes": [] 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "election_epoch": 4, 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: "quorum": [ 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: 0, 2024-01-15T14:36:37.182 INFO:teuthology.orchestra.run.smithi077.stdout: 1, 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: 2 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "quorum_names": [ 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "a", 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "b", 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "c" 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "quorum_age": 117, 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "monmap": { 2024-01-15T14:36:37.183 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 1, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "min_mon_release_name": "reef", 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "num_mons": 3 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap": { 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 34, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "num_osds": 8, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "num_up_osds": 8, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_up_since": 1705329291, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "num_in_osds": 8, 2024-01-15T14:36:37.184 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_in_since": 1705329281, 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "num_remapped_pgs": 0 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap": { 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "pgs_by_state": [ 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "state_name": "active+clean", 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "count": 97 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "num_pgs": 97, 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "num_pools": 3, 2024-01-15T14:36:37.185 INFO:teuthology.orchestra.run.smithi077.stdout: "num_objects": 793, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "data_bytes": 3221998805, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 6668460032, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_avail": 766425653248, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_total": 773094113280, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "write_bytes_sec": 89813584, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "read_op_per_sec": 0, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "write_op_per_sec": 23 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "fsmap": { 2024-01-15T14:36:37.186 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 18, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 2, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "up": 1, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "in": 1, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "by_rank": [ 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "filesystem_id": 2, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "rank": 0, 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "a", 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "up:active", 2024-01-15T14:36:37.187 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 5088 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "up:standby": 3 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "mgrmap": { 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "available": true, 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "num_standbys": 1, 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "modules": [ 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "iostat", 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "nfs", 2024-01-15T14:36:37.188 INFO:teuthology.orchestra.run.smithi077.stdout: "restful" 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "services": {} 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "servicemap": { 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 4, 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "modified": "2024-01-15T14:35:53.029692+0000", 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "services": { 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "daemons": { 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "summary": "", 2024-01-15T14:36:37.189 INFO:teuthology.orchestra.run.smithi077.stdout: "a": { 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "start_epoch": 0, 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "start_stamp": "0.000000", 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 0, 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": {}, 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "task_status": {} 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "b": { 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "start_epoch": 0, 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "start_stamp": "0.000000", 2024-01-15T14:36:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 0, 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": {}, 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "task_status": {} 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "c": { 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "start_epoch": 0, 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "start_stamp": "0.000000", 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 0, 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": {}, 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: "task_status": {} 2024-01-15T14:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "d": { 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "start_epoch": 0, 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "start_stamp": "0.000000", 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 0, 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": {}, 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: "task_status": {} 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.193 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.193 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.193 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_events": {} 2024-01-15T14:36:37.193 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:37.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.183+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f1406b500 msgr2=0x7f5f1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.183+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f1406b500 0x7f5f1406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400d540 tx=0x7f5f24007810 comp rx=0 tx=0).stop 2024-01-15T14:36:37.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.183+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 msgr2=0x7f5f340c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.183+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 0x7f5f340c7680 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c014040 tx=0x7f5f1c004750 comp rx=0 tx=0).stop 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 shutdown_connections 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5f1406b500 0x7f5f1406d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5f3415c070 0x7f5f3409ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5f3406f060 0x7f5f3414bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 --2- 172.21.15.77:0/2470788276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3415bc90 0x7f5f340c7680 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 >> 172.21.15.77:0/2470788276 conn(0x7f5f34073130 msgr2=0x7f5f340b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 shutdown_connections 2024-01-15T14:36:37.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.184+0000 7f5f39c7b700 1 -- 172.21.15.77:0/2470788276 wait complete. 2024-01-15T14:36:37.196 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T14:36:37.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:37.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.323+0000 7f89c9d5a700 1 Processor -- start 2024-01-15T14:36:37.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.323+0000 7f89c9d5a700 1 -- start start 2024-01-15T14:36:37.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c9d5a700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f89c415d1b0 0x7f89c4072630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c9d5a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c40cf900 con 0x7f89c4072250 2024-01-15T14:36:37.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c9d5a700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c40cfbb0 con 0x7f89c4073f90 2024-01-15T14:36:37.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c9d5a700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c415c4e0 con 0x7f89c415d1b0 2024-01-15T14:36:37.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c37fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f89c4072250 0x7f89c40cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45226/0 (socket says 172.21.15.77:45226) 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c37fe700 1 -- 172.21.15.77:0/1531990055 learned_addr learned my addr 172.21.15.77:0/1531990055 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762017196 0 0) 0x7f89c40cf900 con 0x7f89c4072250 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b00035b0 con 0x7f89c4072250 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89bbfff700 1 --2- 172.21.15.77:0/1531990055 >> v2:172.21.15.159:3300/0 conn(0x7f89c415d1b0 0x7f89c4072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4122922134 0 0) 0x7f89b00035b0 con 0x7f89c4072250 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 >> v2:172.21.15.159:3300/0 conn(0x7f89c415d1b0 msgr2=0x7f89c4072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 --2- 172.21.15.77:0/1531990055 >> v2:172.21.15.159:3300/0 conn(0x7f89c415d1b0 0x7f89c4072630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 >> v1:172.21.15.159:6789/0 conn(0x7f89c4073f90 legacy=0x7f89c40743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c415c6d0 con 0x7f89c4072250 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f89b4001670 con 0x7f89c4072250 2024-01-15T14:36:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 >> v1:172.21.15.77:6789/0 conn(0x7f89c4072250 legacy=0x7f89c40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b0003a00 0x7f89b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b0004850 0x7f89b0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0005210 0x7f89b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b000c130 con 0x7f89b0003a00 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b000c2a0 con 0x7f89b0005210 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b000c460 con 0x7f89b0004850 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.324+0000 7f89bbfff700 1 --2- 172.21.15.77:0/1531990055 >> v2:172.21.15.159:3300/0 conn(0x7f89c415d1b0 0x7f89c4072630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b0003a00 0x7f89b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c3fff700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0005210 0x7f89b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89bbfff700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b0004850 0x7f89b0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 -- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b0004850 msgr2=0x7f89b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89b0004850 0x7f89b0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 -- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0005210 msgr2=0x7f89b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0005210 0x7f89b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 -- 172.21.15.77:0/1531990055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00035b0 con 0x7f89b0003a00 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c3fff700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0005210 0x7f89b000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c3fff700 1 -- 172.21.15.77:0/1531990055 reap_dead start 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c37fe700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b0003a00 0x7f89b0003e20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c240 tx=0x7f89b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8e032decadd43d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4007c10 con 0x7f89b0003a00 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b400ce00 con 0x7f89b0003a00 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.325+0000 7f89c2ffd700 1 -- 172.21.15.77:0/1531990055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4005ba0 con 0x7f89b0003a00 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 -- 172.21.15.77:0/1531990055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b0003a00 msgr2=0x7f89b0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b0003a00 0x7f89b0003e20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c240 tx=0x7f89b0005880 comp rx=0 tx=0).stop 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 -- 172.21.15.77:0/1531990055 shutdown_connections 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/1531990055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b0003a00 0x7f89b0003e20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 -- 172.21.15.77:0/1531990055 >> 172.21.15.77:0/1531990055 conn(0x7f89c40cd6b0 msgr2=0x7f89c40cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 -- 172.21.15.77:0/1531990055 shutdown_connections 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.326+0000 7f89c9d5a700 1 -- 172.21.15.77:0/1531990055 wait complete. 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.327+0000 7f89c9d5a700 1 Processor -- start 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.327+0000 7f89c9d5a700 1 -- start start 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0003a00 0x7f89c40ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89c40c85d0 0x7f89c40c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 0x7f89c406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40c8f90 con 0x7f89c406d9a0 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c406e720 con 0x7f89b0003a00 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c406ea00 con 0x7f89c40c85d0 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 0x7f89c406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 0x7f89c406de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38450/0 (socket says 172.21.15.77:38450) 2024-01-15T14:36:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 -- 172.21.15.77:0/3953615554 learned_addr learned my addr 172.21.15.77:0/3953615554 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c37fe700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0003a00 0x7f89c40ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89bbfff700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89c40c85d0 0x7f89c40c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 -- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89c40c85d0 msgr2=0x7f89c40c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89c40c85d0 0x7f89c40c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 -- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0003a00 msgr2=0x7f89c40ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0003a00 0x7f89c40ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 -- 172.21.15.77:0/3953615554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c406ece0 con 0x7f89c406d9a0 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c37fe700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0003a00 0x7f89c40ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c3fff700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 0x7f89c406de50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f89c40b5540 tx=0x7f89bc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc003c10 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc003d70 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c406efc0 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc005310 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.328+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c40cd2d0 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.329+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c4069020 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.330+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f89bc005470 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.330+0000 7f89c0ff9700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f89a406b500 0x7f89a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.330+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89bc089d90 con 0x7f89c406d9a0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.330+0000 7f89c37fe700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f89a406b500 0x7f89a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.331+0000 7f89c37fe700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f89a406b500 0x7f89a406d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f89b4015040 tx=0x7f89b4007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:37.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.335+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89bc054a10 con 0x7f89c406d9a0 2024-01-15T14:36:37.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.567+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f89c40cfe90 con 0x7f89a406b500 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.573+0000 7f89c0ff9700 1 -- 172.21.15.77:0/3953615554 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f89b4007b30 con 0x7f89a406b500 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 0.0, 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "0 B/3 GB", 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "0/3", 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:37.574 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:37.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f89a406b500 msgr2=0x7f89a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f89a406b500 0x7f89a406d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f89b4015040 tx=0x7f89b4007000 comp rx=0 tx=0).stop 2024-01-15T14:36:37.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 msgr2=0x7f89c406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 0x7f89c406de50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f89c40b5540 tx=0x7f89bc00ee00 comp rx=0 tx=0).stop 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 shutdown_connections 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f89a406b500 0x7f89a406d9c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89c406d9a0 0x7f89c406de50 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f89b0003a00 0x7f89c40ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 --2- 172.21.15.77:0/3953615554 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f89c40c85d0 0x7f89c40c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 >> 172.21.15.77:0/3953615554 conn(0x7f89c40cd6b0 msgr2=0x7f89c40d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 shutdown_connections 2024-01-15T14:36:37.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:37.575+0000 7f89c9d5a700 1 -- 172.21.15.77:0/3953615554 wait complete. 2024-01-15T14:36:37.588 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:38.588 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:38.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.727+0000 7fefa29ce700 1 Processor -- start 2024-01-15T14:36:38.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.727+0000 7fefa29ce700 1 -- start start 2024-01-15T14:36:38.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fefa29ce700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fef9c15f8d0 0x7fef9c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fefa29ce700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0b3860 con 0x7fef9c0cffc0 2024-01-15T14:36:38.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fefa29ce700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0b2520 con 0x7fef9c0d03a0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fefa29ce700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c0b2800 con 0x7fef9c15f8d0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fef9c0cffc0 0x7fef9c15f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45236/0 (socket says 172.21.15.77:45236) 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9bfff700 1 -- 172.21.15.77:0/4228648278 learned_addr learned my addr 172.21.15.77:0/4228648278 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/4228648278 >> v2:172.21.15.159:3300/0 conn(0x7fef9c15f8d0 0x7fef9c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059614451 0 0) 0x7fef8c001110 con 0x7fef9c0cffc0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef840035b0 con 0x7fef9c0cffc0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447379915 0 0) 0x7fef840035b0 con 0x7fef9c0cffc0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 >> v2:172.21.15.159:3300/0 conn(0x7fef9c15f8d0 msgr2=0x7fef9c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 --2- 172.21.15.77:0/4228648278 >> v2:172.21.15.159:3300/0 conn(0x7fef9c15f8d0 0x7fef9c0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 >> v1:172.21.15.159:6789/0 conn(0x7fef9c0d03a0 legacy=0x7fef9c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0b2ae0 con 0x7fef9c0cffc0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fef8c001800 con 0x7fef9c0cffc0 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 >> v1:172.21.15.77:6789/0 conn(0x7fef9c0cffc0 legacy=0x7fef9c15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:38.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef84003a00 0x7fef84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef84004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef84005210 0x7fef8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0b2800 con 0x7fef84004850 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8400c1b0 con 0x7fef84003a00 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.728+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8400c370 con 0x7fef84005210 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef84004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9bfff700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef84003a00 0x7fef84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fefa0f6b700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef84005210 0x7fef8400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 -- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef84005210 msgr2=0x7fef8400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef84005210 0x7fef8400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 -- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef84003a00 msgr2=0x7fef84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef84003a00 0x7fef84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 -- 172.21.15.77:0/4228648278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840035b0 con 0x7fef84004850 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fefa0f6b700 1 -- 172.21.15.77:0/4228648278 reap_dead start 2024-01-15T14:36:38.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef84004cd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fef90009b60 tx=0x7fef84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08f861f9b22399d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000e6c0 con 0x7fef84004850 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9000e820 con 0x7fef84004850 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.729+0000 7fef9affd700 1 -- 172.21.15.77:0/4228648278 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000eb90 con 0x7fef84004850 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 -- 172.21.15.77:0/4228648278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 msgr2=0x7fef84004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef84004cd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fef90009b60 tx=0x7fef84005880 comp rx=0 tx=0).stop 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 -- 172.21.15.77:0/4228648278 shutdown_connections 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 --2- 172.21.15.77:0/4228648278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef84004cd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 -- 172.21.15.77:0/4228648278 >> 172.21.15.77:0/4228648278 conn(0x7fef9c0d1e50 msgr2=0x7fef9c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 -- 172.21.15.77:0/4228648278 shutdown_connections 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.730+0000 7fefa29ce700 1 -- 172.21.15.77:0/4228648278 wait complete. 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.731+0000 7fefa29ce700 1 Processor -- start 2024-01-15T14:36:38.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 -- start start 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef9c1536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c153db0 0x7fef9c0cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c0afe60 0x7fef9c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0b0820 con 0x7fef84004850 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0b4c70 con 0x7fef9c153db0 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.732+0000 7fefa29ce700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c79f0 con 0x7fef9c0afe60 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef9c1536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef9c1536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38474/0 (socket says 172.21.15.77:38474) 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 -- 172.21.15.77:0/1031648079 learned_addr learned my addr 172.21.15.77:0/1031648079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c153db0 0x7fef9c0cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fefa0f6b700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c0afe60 0x7fef9c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 -- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c0afe60 msgr2=0x7fef9c0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c0afe60 0x7fef9c0b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 -- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c153db0 msgr2=0x7fef9c0cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c153db0 0x7fef9c0cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 -- 172.21.15.77:0/1031648079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0c7cd0 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef9bfff700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef9c1536b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fef8c007320 tx=0x7fef8c0076e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c007e40 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c007800 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c0b4f30 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c00fa90 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.733+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0c9f70 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.734+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c0d1460 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.735+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fef8c00fbf0 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.735+0000 7fef98ff9700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fef8806b500 0x7fef8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.735+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef8c089d50 con 0x7fef84004850 2024-01-15T14:36:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.735+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fef8806b500 0x7fef8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.735+0000 7fef9b7fe700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fef8806b500 0x7fef8806d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fef9c158420 tx=0x7fef9001e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:38.740+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef8c08e070 con 0x7fef84004850 2024-01-15T14:36:39.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.115+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fef9c0b4af0 con 0x7fef8806b500 2024-01-15T14:36:39.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.126+0000 7fef98ff9700 1 -- 172.21.15.77:0/1031648079 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fef9000ea90 con 0x7fef8806b500 2024-01-15T14:36:39.126 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:39.126 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:39.126 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:39.126 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:39.126 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 0.0, 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "0 B/3 GB", 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "0/3", 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:39.127 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fef8806b500 msgr2=0x7fef8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fef8806b500 0x7fef8806d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fef9c158420 tx=0x7fef9001e040 comp rx=0 tx=0).stop 2024-01-15T14:36:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 msgr2=0x7fef9c1536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef9c1536b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fef8c007320 tx=0x7fef8c0076e0 comp rx=0 tx=0).stop 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 shutdown_connections 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fef8806b500 0x7fef8806d9c0 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fef9c158420 tx=0x7fef9001e040 comp rx=0 tx=0).stop 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fef9c0afe60 0x7fef9c0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef84004850 0x7fef9c1536b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 --2- 172.21.15.77:0/1031648079 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fef9c153db0 0x7fef9c0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 >> 172.21.15.77:0/1031648079 conn(0x7fef9c0d1e50 msgr2=0x7fef9c070560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 shutdown_connections 2024-01-15T14:36:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:39.132+0000 7fefa29ce700 1 -- 172.21.15.77:0/1031648079 wait complete. 2024-01-15T14:36:39.160 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:40.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:40.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.372+0000 7f6148f18700 1 Processor -- start 2024-01-15T14:36:40.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.372+0000 7f6148f18700 1 -- start start 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6148f18700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61441626c0 0x7f614406bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6148f18700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f614406c010 0x7f61440c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6148f18700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6144162aa0 con 0x7f61440b42a0 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6148f18700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614406cc20 con 0x7f614406c010 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6148f18700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614406cec0 con 0x7f61441626c0 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6141d9b700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f61440b42a0 0x7f61440cc220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:59120/0 (socket says 172.21.15.77:59120) 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6141d9b700 1 -- 172.21.15.77:0/2132588394 learned_addr learned my addr 172.21.15.77:0/2132588394 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f6142d9d700 1 --2- 172.21.15.77:0/2132588394 >> v2:172.21.15.159:3300/0 conn(0x7f614406c010 0x7f61440c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.373+0000 7f614259c700 1 --2- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61441626c0 0x7f614406bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.374+0000 7f6142d9d700 1 -- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61441626c0 msgr2=0x7f614406bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.374+0000 7f6142d9d700 1 --2- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61441626c0 0x7f614406bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.374+0000 7f6142d9d700 1 -- 172.21.15.77:0/2132588394 >> v1:172.21.15.159:6789/0 conn(0x7f61440b42a0 legacy=0x7f61440cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.374+0000 7f614159a700 1 -- 172.21.15.77:0/2132588394 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027747460 0 0) 0x7f6144162aa0 con 0x7f61440b42a0 2024-01-15T14:36:40.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.374+0000 7f6142d9d700 1 -- 172.21.15.77:0/2132588394 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614406aff0 con 0x7f614406c010 2024-01-15T14:36:40.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.375+0000 7f6142d9d700 1 --2- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f614406c010 0x7f61440c4ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f613c009d90 tx=0x7f613c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0ef8d9f9ac2f8f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:40.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.375+0000 7f614159a700 1 -- 172.21.15.77:0/2132588394 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00e070 con 0x7f614406c010 2024-01-15T14:36:40.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.375+0000 7f614159a700 1 -- 172.21.15.77:0/2132588394 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c00bbd0 con 0x7f614406c010 2024-01-15T14:36:40.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.375+0000 7f614159a700 1 -- 172.21.15.77:0/2132588394 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c012430 con 0x7f614406c010 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.376+0000 7f6148f18700 1 -- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f614406c010 msgr2=0x7f61440c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.376+0000 7f6148f18700 1 --2- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f614406c010 0x7f61440c4ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f613c009d90 tx=0x7f613c00b330 comp rx=0 tx=0).stop 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.378+0000 7f6148f18700 1 -- 172.21.15.77:0/2132588394 shutdown_connections 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.378+0000 7f6148f18700 1 --2- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f614406c010 0x7f61440c4ba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.378+0000 7f6148f18700 1 --2- 172.21.15.77:0/2132588394 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f61441626c0 0x7f614406bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.378+0000 7f6148f18700 1 -- 172.21.15.77:0/2132588394 >> 172.21.15.77:0/2132588394 conn(0x7f61440b3780 msgr2=0x7f61440b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.378+0000 7f6148f18700 1 -- 172.21.15.77:0/2132588394 shutdown_connections 2024-01-15T14:36:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.378+0000 7f6148f18700 1 -- 172.21.15.77:0/2132588394 wait complete. 2024-01-15T14:36:40.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.381+0000 7f6148f18700 1 Processor -- start 2024-01-15T14:36:40.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.382+0000 7f6148f18700 1 -- start start 2024-01-15T14:36:40.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.382+0000 7f6148f18700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 0x7f614415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:40.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.382+0000 7f6148f18700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61440b42a0 0x7f6144160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:40.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6148f18700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 0x7f6144161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6148f18700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440d0610 con 0x7f61440b42a0 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6148f18700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440cf600 con 0x7f61441626c0 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6148f18700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440d0180 con 0x7f614406c010 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 0x7f614415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6142d9d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 0x7f6144161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 0x7f614415c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:45344/0 (socket says 172.21.15.77:45344) 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6142d9d700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 0x7f6144161200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:48616/0 (socket says 172.21.15.77:48616) 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 -- 172.21.15.77:0/3002318464 learned_addr learned my addr 172.21.15.77:0/3002318464 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 -- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 msgr2=0x7f6144161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 0x7f6144161200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 -- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61440b42a0 msgr2=0x7f6144160cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61440b42a0 0x7f6144160cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f614259c700 1 -- 172.21.15.77:0/3002318464 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c009a70 con 0x7f614406c010 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.383+0000 7f6142d9d700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 0x7f6144161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:36:40.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.384+0000 7f614259c700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 0x7f614415c4a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6134007f90 tx=0x7f613400da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:40.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.384+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134036640 con 0x7f614406c010 2024-01-15T14:36:40.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.384+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614415c9e0 con 0x7f614406c010 2024-01-15T14:36:40.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.384+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61340367a0 con 0x7f614406c010 2024-01-15T14:36:40.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.384+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134036b30 con 0x7f614406c010 2024-01-15T14:36:40.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.384+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61440cfb30 con 0x7f614406c010 2024-01-15T14:36:40.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.386+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6144069020 con 0x7f614406c010 2024-01-15T14:36:40.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.386+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mon.2 v2:172.21.15.159:3301/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f6134036c90 con 0x7f614406c010 2024-01-15T14:36:40.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.386+0000 7f6132ffd700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f612406b500 0x7f612406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:40.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.387+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mon.2 v2:172.21.15.159:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61340ac260 con 0x7f614406c010 2024-01-15T14:36:40.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.387+0000 7f6141d9b700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f612406b500 0x7f612406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:40.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.387+0000 7f6141d9b700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f612406b500 0x7f612406d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6138007fd0 tx=0x7f6138007940 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:40.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.393+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mon.2 v2:172.21.15.159:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6134076e70 con 0x7f614406c010 2024-01-15T14:36:40.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.662+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f614406b5b0 con 0x7f612406b500 2024-01-15T14:36:40.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.669+0000 7f6132ffd700 1 -- 172.21.15.77:0/3002318464 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f6138001310 con 0x7f612406b500 2024-01-15T14:36:40.669 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:40.669 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:40.669 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:40.669 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 33.33333333333333, 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "1/3", 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:40.670 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f612406b500 msgr2=0x7f612406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f612406b500 0x7f612406d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6138007fd0 tx=0x7f6138007940 comp rx=0 tx=0).stop 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 msgr2=0x7f614415c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 0x7f614415c4a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6134007f90 tx=0x7f613400da30 comp rx=0 tx=0).stop 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 shutdown_connections 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f612406b500 0x7f612406d9c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f614406c010 0x7f614415c4a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f61441626c0 0x7f6144161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 --2- 172.21.15.77:0/3002318464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61440b42a0 0x7f6144160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 >> 172.21.15.77:0/3002318464 conn(0x7f61440b3780 msgr2=0x7f61440b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 shutdown_connections 2024-01-15T14:36:40.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:40.671+0000 7f6148f18700 1 -- 172.21.15.77:0/3002318464 wait complete. 2024-01-15T14:36:40.683 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:41.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:41.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.824+0000 7f15789db700 1 Processor -- start 2024-01-15T14:36:41.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.824+0000 7f15789db700 1 -- start start 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f15789db700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f157406db80 0x7f1574156400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f15789db700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157406f5e0 con 0x7f157406c710 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f15789db700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157406f8c0 con 0x7f15740cf740 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f15789db700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157406fba0 con 0x7f157406db80 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f1571d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f157406c710 0x7f1574155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48950/0 (socket says 172.21.15.77:48950) 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f1571d9b700 1 -- 172.21.15.77:0/2786937798 learned_addr learned my addr 172.21.15.77:0/2786937798 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:41.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194684095 0 0) 0x7f157406f5e0 con 0x7f157406c710 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f155c0035b0 con 0x7f157406c710 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f1572d9d700 1 --2- 172.21.15.77:0/2786937798 >> v2:172.21.15.159:3300/0 conn(0x7f157406db80 0x7f1574156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2083314999 0 0) 0x7f155c0035b0 con 0x7f157406c710 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 >> v2:172.21.15.159:3300/0 conn(0x7f157406db80 msgr2=0x7f1574156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 --2- 172.21.15.77:0/2786937798 >> v2:172.21.15.159:3300/0 conn(0x7f157406db80 0x7f1574156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 >> v1:172.21.15.159:6789/0 conn(0x7f15740cf740 legacy=0x7f157406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740728d0 con 0x7f157406c710 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.825+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f1568001670 con 0x7f157406c710 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 >> v1:172.21.15.77:6789/0 conn(0x7f157406c710 legacy=0x7f1574155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c003a00 0x7f155c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c004850 0x7f155c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f155c005210 0x7f155c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157406fba0 con 0x7f155c003a00 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c00c1b0 con 0x7f155c005210 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c00c370 con 0x7f155c004850 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c003a00 0x7f155c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f1572d9d700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f155c005210 0x7f155c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f1571d9b700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c004850 0x7f155c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 -- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c004850 msgr2=0x7f155c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c004850 0x7f155c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 -- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f155c005210 msgr2=0x7f155c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f155c005210 0x7f155c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:41.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 -- 172.21.15.77:0/2786937798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0035b0 con 0x7f155c003a00 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f1572d9d700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f155c005210 0x7f155c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f1572d9d700 1 -- 172.21.15.77:0/2786937798 reap_dead start 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157259c700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c003a00 0x7f155c003e20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1564001800 tx=0x7f155c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc58f22b75a6180e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15640012f0 con 0x7f155c003a00 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1564005310 con 0x7f155c003a00 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.826+0000 7f157159a700 1 -- 172.21.15.77:0/2786937798 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1564006860 con 0x7f155c003a00 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 -- 172.21.15.77:0/2786937798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c003a00 msgr2=0x7f155c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c003a00 0x7f155c003e20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1564001800 tx=0x7f155c005880 comp rx=0 tx=0).stop 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 -- 172.21.15.77:0/2786937798 shutdown_connections 2024-01-15T14:36:41.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 --2- 172.21.15.77:0/2786937798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c003a00 0x7f155c003e20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 -- 172.21.15.77:0/2786937798 >> 172.21.15.77:0/2786937798 conn(0x7f15740ce500 msgr2=0x7f1574071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 -- 172.21.15.77:0/2786937798 shutdown_connections 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.827+0000 7f15789db700 1 -- 172.21.15.77:0/2786937798 wait complete. 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.828+0000 7f15789db700 1 Processor -- start 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.828+0000 7f15789db700 1 -- start start 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f15789db700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c003a00 0x7f1574156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f15789db700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15741595d0 0x7f15741f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f15789db700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 0x7f15741f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f15789db700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1574156940 con 0x7f15741f6030 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f15789db700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157415db60 con 0x7f15741595d0 2024-01-15T14:36:41.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f15789db700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1574148d90 con 0x7f155c003a00 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f1572d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 0x7f15741f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f1572d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 0x7f15741f64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33410/0 (socket says 172.21.15.77:33410) 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f1572d9d700 1 -- 172.21.15.77:0/1611994531 learned_addr learned my addr 172.21.15.77:0/1611994531 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f157259c700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c003a00 0x7f1574156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.829+0000 7f1571d9b700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15741595d0 0x7f15741f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1572d9d700 1 -- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c003a00 msgr2=0x7f1574156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1572d9d700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c003a00 0x7f1574156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1572d9d700 1 -- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15741595d0 msgr2=0x7f15741f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1572d9d700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15741595d0 0x7f15741f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1572d9d700 1 -- 172.21.15.77:0/1611994531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574159d60 con 0x7f15741f6030 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1572d9d700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 0x7f15741f64b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1574074fa0 tx=0x7f156c0101c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c01b070 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c00f040 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15741f21d0 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c00e050 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.830+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157415d880 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.831+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574069020 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.831+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f156c010a00 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.832+0000 7f1562ffd700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f155406b530 0x7f155406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.832+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f156c089fd0 con 0x7f15741f6030 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.832+0000 7f157259c700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f155406b530 0x7f155406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:41.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.832+0000 7f157259c700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f155406b530 0x7f155406d9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1564001800 tx=0x7f1564009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:41.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:41.836+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f156c054b40 con 0x7f15741f6030 2024-01-15T14:36:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.069+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f15740cfb20 con 0x7f155406b530 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.074+0000 7f1562ffd700 1 -- 172.21.15.77:0/1611994531 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f1564005480 con 0x7f155406b530 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:42.075 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 33.33333333333333, 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "1/3", 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:42.076 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f155406b530 msgr2=0x7f155406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f155406b530 0x7f155406d9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1564001800 tx=0x7f1564009030 comp rx=0 tx=0).stop 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 msgr2=0x7f15741f64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 0x7f15741f64b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1574074fa0 tx=0x7f156c0101c0 comp rx=0 tx=0).stop 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 shutdown_connections 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f155406b530 0x7f155406d9f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f155c003a00 0x7f1574156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15741f6030 0x7f15741f64b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 --2- 172.21.15.77:0/1611994531 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f15741595d0 0x7f15741f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 >> 172.21.15.77:0/1611994531 conn(0x7f15740ce500 msgr2=0x7f1574152900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 shutdown_connections 2024-01-15T14:36:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:42.076+0000 7f15789db700 1 -- 172.21.15.77:0/1611994531 wait complete. 2024-01-15T14:36:42.089 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:43.090 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.300+0000 7f92c91fc700 1 Processor -- start 2024-01-15T14:36:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.300+0000 7f92c91fc700 1 -- start start 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c91fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92c4151260 0x7f92c415b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c91fc700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f92c4151640 0x7f92c40b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c91fc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c406cb20 con 0x7f92c406d6c0 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c91fc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c40b40c0 con 0x7f92c4151260 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c91fc700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c415b980 con 0x7f92c4151640 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c2d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92c4151260 0x7f92c415b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c2d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92c4151260 0x7f92c415b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33426/0 (socket says 172.21.15.77:33426) 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c2d9d700 1 -- 172.21.15.77:0/1351673176 learned_addr learned my addr 172.21.15.77:0/1351673176 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.302+0000 7f92c359e700 1 --1- 172.21.15.77:0/1351673176 >> v1:172.21.15.77:6789/0 conn(0x7f92c406d6c0 0x7f92c415dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48956/0 (socket says 172.21.15.77:48956) 2024-01-15T14:36:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c259c700 1 --2- 172.21.15.77:0/1351673176 >> v2:172.21.15.159:3300/0 conn(0x7f92c4151640 0x7f92c40b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c1d9b700 1 -- 172.21.15.77:0/1351673176 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448762811 0 0) 0x7f92c406cb20 con 0x7f92c406d6c0 2024-01-15T14:36:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c2d9d700 1 -- 172.21.15.77:0/1351673176 >> v2:172.21.15.159:3300/0 conn(0x7f92c4151640 msgr2=0x7f92c40b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c2d9d700 1 --2- 172.21.15.77:0/1351673176 >> v2:172.21.15.159:3300/0 conn(0x7f92c4151640 0x7f92c40b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c2d9d700 1 -- 172.21.15.77:0/1351673176 >> v1:172.21.15.77:6789/0 conn(0x7f92c406d6c0 legacy=0x7f92c415dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c2d9d700 1 -- 172.21.15.77:0/1351673176 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c406cd70 con 0x7f92c4151260 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c2d9d700 1 --2- 172.21.15.77:0/1351673176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c415b440 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f92c406f690 tx=0x7f92b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea56e06b31f76270 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c1d9b700 1 -- 172.21.15.77:0/1351673176 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400b870 con 0x7f92c4151260 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c1d9b700 1 -- 172.21.15.77:0/1351673176 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400b9d0 con 0x7f92c4151260 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.303+0000 7f92c1d9b700 1 -- 172.21.15.77:0/1351673176 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400bd60 con 0x7f92c4151260 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 -- 172.21.15.77:0/1351673176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 msgr2=0x7f92c415b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 --2- 172.21.15.77:0/1351673176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c415b440 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f92c406f690 tx=0x7f92b4009bd0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 -- 172.21.15.77:0/1351673176 shutdown_connections 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 --2- 172.21.15.77:0/1351673176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c415b440 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 --2- 172.21.15.77:0/1351673176 >> v2:172.21.15.159:3300/0 conn(0x7f92c4151640 0x7f92c40b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 -- 172.21.15.77:0/1351673176 >> 172.21.15.77:0/1351673176 conn(0x7f92c40b1760 msgr2=0x7f92c40b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 -- 172.21.15.77:0/1351673176 shutdown_connections 2024-01-15T14:36:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.304+0000 7f92c91fc700 1 -- 172.21.15.77:0/1351673176 wait complete. 2024-01-15T14:36:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.306+0000 7f92c91fc700 1 Processor -- start 2024-01-15T14:36:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.307+0000 7f92c91fc700 1 -- start start 2024-01-15T14:36:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c91fc700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c406d6c0 0x7f92c4153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c91fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c41f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c91fc700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4151640 0x7f92c41f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c91fc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c4154410 con 0x7f92c4151260 2024-01-15T14:36:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c91fc700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c414b9e0 con 0x7f92c406d6c0 2024-01-15T14:36:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c91fc700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c4154660 con 0x7f92c4151640 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c41f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c41f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33434/0 (socket says 172.21.15.77:33434) 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 -- 172.21.15.77:0/258196084 learned_addr learned my addr 172.21.15.77:0/258196084 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c2d9d700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c406d6c0 0x7f92c4153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 -- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4151640 msgr2=0x7f92c41f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4151640 0x7f92c41f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 -- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c406d6c0 msgr2=0x7f92c4153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c406d6c0 0x7f92c4153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.308+0000 7f92c259c700 1 -- 172.21.15.77:0/258196084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009550 con 0x7f92c4151260 2024-01-15T14:36:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.309+0000 7f92c259c700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c41f19d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f92b800e840 tx=0x7f92b800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.309+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8013610 con 0x7f92c4151260 2024-01-15T14:36:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.309+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b8013770 con 0x7f92c4151260 2024-01-15T14:36:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.309+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c414b670 con 0x7f92c4151260 2024-01-15T14:36:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.309+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8013b00 con 0x7f92c4151260 2024-01-15T14:36:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.309+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c414be70 con 0x7f92c4151260 2024-01-15T14:36:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.310+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c4069020 con 0x7f92c4151260 2024-01-15T14:36:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.311+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f92b8013c60 con 0x7f92c4151260 2024-01-15T14:36:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.311+0000 7f92ab7fe700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b500 0x7f92b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.311+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92b8089bd0 con 0x7f92c4151260 2024-01-15T14:36:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.313+0000 7f92c2d9d700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b500 0x7f92b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.314+0000 7f92c2d9d700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b500 0x7f92b006d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f92b4000c00 tx=0x7f92b4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.317+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92b8054740 con 0x7f92c4151260 2024-01-15T14:36:43.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.689+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f92c40cdcb0 con 0x7f92b006b500 2024-01-15T14:36:43.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.702+0000 7f92ab7fe700 1 -- 172.21.15.77:0/258196084 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f92c40cdcb0 con 0x7f92b006b500 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:43.703 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 33.33333333333333, 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "1/3", 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:43.704 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b500 msgr2=0x7f92b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b500 0x7f92b006d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f92b4000c00 tx=0x7f92b4008030 comp rx=0 tx=0).stop 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 msgr2=0x7f92c41f19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c41f19d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f92b800e840 tx=0x7f92b800ec00 comp rx=0 tx=0).stop 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 shutdown_connections 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f92b006b500 0x7f92b006d9c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f92c4151640 0x7f92c41f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f92c406d6c0 0x7f92c4153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 --2- 172.21.15.77:0/258196084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92c4151260 0x7f92c41f19d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:43.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 >> 172.21.15.77:0/258196084 conn(0x7f92c40b1760 msgr2=0x7f92c40b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:43.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 shutdown_connections 2024-01-15T14:36:43.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:43.704+0000 7f92c91fc700 1 -- 172.21.15.77:0/258196084 wait complete. 2024-01-15T14:36:43.735 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:44.736 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:44.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.861+0000 7f35c5cc0700 1 Processor -- start 2024-01-15T14:36:44.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.861+0000 7f35c5cc0700 1 -- start start 2024-01-15T14:36:44.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35c5cc0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35c0161690 0x7f35c00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:44.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35c5cc0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161a70 0x7f35c00d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:44.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35c5cc0700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00d0a70 con 0x7f35c00d1730 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35c5cc0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c00d1c10 con 0x7f35c0161690 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35c5cc0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c00b2ab0 con 0x7f35c0161a70 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35bf7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35c0161690 0x7f35c00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35bf7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35c0161690 0x7f35c00d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33450/0 (socket says 172.21.15.77:33450) 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.862+0000 7f35bf7fe700 1 -- 172.21.15.77:0/168975522 learned_addr learned my addr 172.21.15.77:0/168975522 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35beffd700 1 --2- 172.21.15.77:0/168975522 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161a70 0x7f35c00d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35bf7fe700 1 -- 172.21.15.77:0/168975522 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161a70 msgr2=0x7f35c00d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35bf7fe700 1 --2- 172.21.15.77:0/168975522 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161a70 0x7f35c00d2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35bf7fe700 1 -- 172.21.15.77:0/168975522 >> v1:172.21.15.159:6789/0 conn(0x7f35c00d1730 legacy=0x7f35c009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35bf7fe700 1 -- 172.21.15.77:0/168975522 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c00b2d90 con 0x7f35c0161690 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35bf7fe700 1 --2- 172.21.15.77:0/168975522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161690 0x7f35c00d0f20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f35a80098f0 tx=0x7f35a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376808d9c09ad926 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35be7fc700 1 -- 172.21.15.77:0/168975522 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8004230 con 0x7f35c0161690 2024-01-15T14:36:44.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35be7fc700 1 -- 172.21.15.77:0/168975522 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8004390 con 0x7f35c0161690 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35be7fc700 1 -- 172.21.15.77:0/168975522 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a80036a0 con 0x7f35c0161690 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35c5cc0700 1 -- 172.21.15.77:0/168975522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161690 msgr2=0x7f35c00d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/168975522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161690 0x7f35c00d0f20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f35a80098f0 tx=0x7f35a8009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35c5cc0700 1 -- 172.21.15.77:0/168975522 shutdown_connections 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/168975522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161690 0x7f35c00d0f20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/168975522 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161a70 0x7f35c00d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.863+0000 7f35c5cc0700 1 -- 172.21.15.77:0/168975522 >> 172.21.15.77:0/168975522 conn(0x7f35c00d1e50 msgr2=0x7f35c00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.864+0000 7f35c5cc0700 1 -- 172.21.15.77:0/168975522 shutdown_connections 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.864+0000 7f35c5cc0700 1 -- 172.21.15.77:0/168975522 wait complete. 2024-01-15T14:36:44.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.865+0000 7f35c5cc0700 1 Processor -- start 2024-01-15T14:36:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.865+0000 7f35c5cc0700 1 -- start start 2024-01-15T14:36:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35c00d1730 0x7f35c00c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161690 0x7f35c015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 0x7f35c00af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c00aeef0 con 0x7f35c0161a70 2024-01-15T14:36:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c00c6c70 con 0x7f35c00d1730 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c00c6f50 con 0x7f35c0161690 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 0x7f35c00af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 0x7f35c00af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33458/0 (socket says 172.21.15.77:33458) 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 -- 172.21.15.77:0/355214018 learned_addr learned my addr 172.21.15.77:0/355214018 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bf7fe700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35c00d1730 0x7f35c00c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35beffd700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161690 0x7f35c015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 -- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161690 msgr2=0x7f35c015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161690 0x7f35c015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 -- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35c00d1730 msgr2=0x7f35c00c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35c00d1730 0x7f35c00c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 -- 172.21.15.77:0/355214018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a8009550 con 0x7f35c0161a70 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35bffff700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 0x7f35c00af320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b9e0 tx=0x7f35b800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:44.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8003c00 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b8003da0 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b800d920 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c00c7230 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.866+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c00c7880 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.867+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c0069020 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.868+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f35b800da80 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.868+0000 7f35b7fff700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f35a006b500 0x7f35a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.868+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35b8011070 con 0x7f35c0161a70 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.868+0000 7f35bf7fe700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f35a006b500 0x7f35a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:44.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.869+0000 7f35bf7fe700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f35a006b500 0x7f35a006d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009130 tx=0x7f35a8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:44.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:44.873+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35b80548f0 con 0x7f35c0161a70 2024-01-15T14:36:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.112+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f35c009b110 con 0x7f35a006b500 2024-01-15T14:36:45.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.117+0000 7f35b7fff700 1 -- 172.21.15.77:0/355214018 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f35a8003720 con 0x7f35a006b500 2024-01-15T14:36:45.118 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 66.66666666666666, 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "2/3", 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:45.119 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:45.120 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:45.120 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:45.120 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f35a006b500 msgr2=0x7f35a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f35a006b500 0x7f35a006d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009130 tx=0x7f35a8008030 comp rx=0 tx=0).stop 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 msgr2=0x7f35c00af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 0x7f35c00af320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b9e0 tx=0x7f35b800bcf0 comp rx=0 tx=0).stop 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 shutdown_connections 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f35a006b500 0x7f35a006d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35c0161a70 0x7f35c00af320 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f35c00d1730 0x7f35c00c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:45.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 --2- 172.21.15.77:0/355214018 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f35c0161690 0x7f35c015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 >> 172.21.15.77:0/355214018 conn(0x7f35c00d1e50 msgr2=0x7f35c00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 shutdown_connections 2024-01-15T14:36:45.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:45.120+0000 7f35c5cc0700 1 -- 172.21.15.77:0/355214018 wait complete. 2024-01-15T14:36:45.133 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:46.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:46.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.321+0000 7fa502373700 1 Processor -- start 2024-01-15T14:36:46.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.321+0000 7fa502373700 1 -- start start 2024-01-15T14:36:46.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.323+0000 7fa502373700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fa4fc15ee20 0x7fa4fc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.323+0000 7fa502373700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc0b5600 con 0x7fa4fc06f580 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.323+0000 7fa502373700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc06fa10 con 0x7fa4fc158600 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.323+0000 7fa502373700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc06e4e0 con 0x7fa4fc15ee20 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4fb7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa4fc06f580 0x7fa4fc15e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48966/0 (socket says 172.21.15.77:48966) 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4fb7fe700 1 -- 172.21.15.77:0/1489146454 learned_addr learned my addr 172.21.15.77:0/1489146454 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587452367 0 0) 0x7fa4fc0b5600 con 0x7fa4fc06f580 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa500910700 1 --2- 172.21.15.77:0/1489146454 >> v2:172.21.15.159:3300/0 conn(0x7fa4fc15ee20 0x7fa4fc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e40035b0 con 0x7fa4fc06f580 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638806716 0 0) 0x7fa4e40035b0 con 0x7fa4fc06f580 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 >> v2:172.21.15.159:3300/0 conn(0x7fa4fc15ee20 msgr2=0x7fa4fc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 --2- 172.21.15.77:0/1489146454 >> v2:172.21.15.159:3300/0 conn(0x7fa4fc15ee20 0x7fa4fc06aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 >> v1:172.21.15.159:6789/0 conn(0x7fa4fc158600 legacy=0x7fa4fc0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc06e7c0 con 0x7fa4fc06f580 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.324+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7fa4f0001670 con 0x7fa4fc06f580 2024-01-15T14:36:46.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 >> v1:172.21.15.77:6789/0 conn(0x7fa4fc06f580 legacy=0x7fa4fc15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4e4004850 0x7fa4e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4e4005210 0x7fa4e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc06e4e0 con 0x7fa4e4003a00 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e400c1b0 con 0x7fa4e4004850 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e400c370 con 0x7fa4e4005210 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1489146454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4e4005210 msgr2=0x7fa4e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4e4005210 0x7fa4e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1489146454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4e4004850 msgr2=0x7fa4e4004cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4e4004850 0x7fa4e4004cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1489146454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e40035b0 con 0x7fa4e4003a00 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa500910700 1 -- 172.21.15.77:0/1489146454 reap_dead start 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4e4003e20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0017c0 tx=0x7fa4e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55449eeec32b9c26 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:46.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0052d0 con 0x7fa4e4003a00 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec005450 con 0x7fa4e4003a00 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.325+0000 7fa4faffd700 1 -- 172.21.15.77:0/1489146454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0038c0 con 0x7fa4e4003a00 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 -- 172.21.15.77:0/1489146454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 msgr2=0x7fa4e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4e4003e20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0017c0 tx=0x7fa4e4005880 comp rx=0 tx=0).stop 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 -- 172.21.15.77:0/1489146454 shutdown_connections 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 --2- 172.21.15.77:0/1489146454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4e4003e20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 -- 172.21.15.77:0/1489146454 >> 172.21.15.77:0/1489146454 conn(0x7fa4fc0b3250 msgr2=0x7fa4fc0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 -- 172.21.15.77:0/1489146454 shutdown_connections 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.326+0000 7fa502373700 1 -- 172.21.15.77:0/1489146454 wait complete. 2024-01-15T14:36:46.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.329+0000 7fa502373700 1 Processor -- start 2024-01-15T14:36:46.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.330+0000 7fa502373700 1 -- start start 2024-01-15T14:36:46.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.332+0000 7fa502373700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4fc144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa502373700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4fc13fcc0 0x7fa4fc140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa502373700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4fc14f400 0x7fa4fc14f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa502373700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc0cb510 con 0x7fa4e4003a00 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa502373700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc150c60 con 0x7fa4fc13fcc0 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa502373700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc140680 con 0x7fa4fc14f400 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa4fbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4fc144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa4fbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4fc144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33478/0 (socket says 172.21.15.77:33478) 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1367150999 learned_addr learned my addr 172.21.15.77:0/1367150999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:46.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa500910700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4fc14f400 0x7fa4fc14f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:46.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.333+0000 7fa4fb7fe700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4fc13fcc0 0x7fa4fc140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:46.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4fc14f400 msgr2=0x7fa4fc14f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4fc14f400 0x7fa4fc14f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4fc13fcc0 msgr2=0x7fa4fc140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4fc13fcc0 0x7fa4fc140140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4fbfff700 1 -- 172.21.15.77:0/1367150999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc14fda0 con 0x7fa4e4003a00 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4fbfff700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4fc144070 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec001db0 tx=0x7fa4ec001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.334+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec001290 con 0x7fa4e4003a00 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.335+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc150050 con 0x7fa4e4003a00 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.335+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc150610 con 0x7fa4e4003a00 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.335+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec007040 con 0x7fa4e4003a00 2024-01-15T14:36:46.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.335+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0073d0 con 0x7fa4e4003a00 2024-01-15T14:36:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.338+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc0052f0 con 0x7fa4e4003a00 2024-01-15T14:36:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.340+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fa4ec007530 con 0x7fa4e4003a00 2024-01-15T14:36:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.341+0000 7fa4f8ff9700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa4e806b500 0x7fa4e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.341+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4ec08a670 con 0x7fa4e4003a00 2024-01-15T14:36:46.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.344+0000 7fa4fb7fe700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa4e806b500 0x7fa4e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:46.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.349+0000 7fa4fb7fe700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa4e806b500 0x7fa4e806d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0001820 tx=0x7fa4f00015f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:46.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.351+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4ec055410 con 0x7fa4e4003a00 2024-01-15T14:36:46.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.744+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4dc000bc0 con 0x7fa4e806b500 2024-01-15T14:36:46.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.758+0000 7fa4f8ff9700 1 -- 172.21.15.77:0/1367150999 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fa4dc000bc0 con 0x7fa4e806b500 2024-01-15T14:36:46.762 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "in-progress", 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "source": { 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "volume": "cephfs", 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "subvolume": "sv1", 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot": "ss1" 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "progress_report": { 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "percentage completed": 66.66666666666666, 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "regfiles copied": "2/3", 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "subdirs copied": "0/1", 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: "symlinks copied": "0/0" 2024-01-15T14:36:46.763 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:46.764 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:46.764 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa4e806b500 msgr2=0x7fa4e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa4e806b500 0x7fa4e806d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0001820 tx=0x7fa4f00015f0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 msgr2=0x7fa4fc144070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4fc144070 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec001db0 tx=0x7fa4ec001120 comp rx=0 tx=0).stop 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 shutdown_connections 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fa4e806b500 0x7fa4e806d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fa4fc14f400 0x7fa4fc14f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4e4003a00 0x7fa4fc144070 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 --2- 172.21.15.77:0/1367150999 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fa4fc13fcc0 0x7fa4fc140140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 >> 172.21.15.77:0/1367150999 conn(0x7fa4fc0b3250 msgr2=0x7fa4fc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 shutdown_connections 2024-01-15T14:36:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:46.767+0000 7fa502373700 1 -- 172.21.15.77:0/1367150999 wait complete. 2024-01-15T14:36:46.790 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T14:36:47.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T14:36:47.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.942+0000 7fb9ee69f700 1 Processor -- start 2024-01-15T14:36:47.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.942+0000 7fb9ee69f700 1 -- start start 2024-01-15T14:36:47.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ee69f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e80b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ee69f700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 0x7fb9e8074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ee69f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e80ae300 con 0x7fb9e80d0630 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ee69f700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e806e350 con 0x7fb9e8073f90 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ee69f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e8074960 con 0x7fb9e815c030 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e80b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ecc3c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 0x7fb9e8074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e80b0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3301/0 says I am v2:172.21.15.77:45422/0 (socket says 172.21.15.77:45422) 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e77fe700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb9e80d0630 0x7fb9e806dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:59160/0 (socket says 172.21.15.77:59160) 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ecc3c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 0x7fb9e8074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:48714/0 (socket says 172.21.15.77:48714) 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 -- 172.21.15.77:0/3883339643 learned_addr learned my addr 172.21.15.77:0/3883339643 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 -- 172.21.15.77:0/3883339643 >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 msgr2=0x7fb9e8074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 --2- 172.21.15.77:0/3883339643 >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 0x7fb9e8074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 -- 172.21.15.77:0/3883339643 >> v1:172.21.15.159:6789/0 conn(0x7fb9e80d0630 legacy=0x7fb9e806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:47.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9e7fff700 1 -- 172.21.15.77:0/3883339643 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e815fb00 con 0x7fb9e815c030 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.943+0000 7fb9ecc3c700 1 --2- 172.21.15.77:0/3883339643 >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 0x7fb9e8074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9e7fff700 1 --2- 172.21.15.77:0/3883339643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e80b0220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0098f0 tx=0x7fb9dc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3f93f3f116c4b02 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9e6ffd700 1 -- 172.21.15.77:0/3883339643 <== mon.2 v2:172.21.15.159:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc011830 con 0x7fb9e815c030 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9e6ffd700 1 -- 172.21.15.77:0/3883339643 <== mon.2 v2:172.21.15.159:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc011990 con 0x7fb9e815c030 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9e6ffd700 1 -- 172.21.15.77:0/3883339643 <== mon.2 v2:172.21.15.159:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc011d20 con 0x7fb9e815c030 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 -- 172.21.15.77:0/3883339643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 msgr2=0x7fb9e80b0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/3883339643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e80b0220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0098f0 tx=0x7fb9dc009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 -- 172.21.15.77:0/3883339643 shutdown_connections 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/3883339643 >> v2:172.21.15.159:3300/0 conn(0x7fb9e8073f90 0x7fb9e8074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/3883339643 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e80b0220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 -- 172.21.15.77:0/3883339643 >> 172.21.15.77:0/3883339643 conn(0x7fb9e80d0c00 msgr2=0x7fb9e80d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 -- 172.21.15.77:0/3883339643 shutdown_connections 2024-01-15T14:36:47.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.944+0000 7fb9ee69f700 1 -- 172.21.15.77:0/3883339643 wait complete. 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.945+0000 7fb9ee69f700 1 Processor -- start 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 -- start start 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9e8073f90 0x7fb9e80b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 0x7fb9e81f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e809c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e815dcc0 con 0x7fb9e80d0630 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e815b4d0 con 0x7fb9e8073f90 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.946+0000 7fb9ee69f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e81f4cc0 con 0x7fb9e815c030 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 0x7fb9e81f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 0x7fb9e81f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33486/0 (socket says 172.21.15.77:33486) 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 -- 172.21.15.77:0/256278604 learned_addr learned my addr 172.21.15.77:0/256278604 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e7fff700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9e8073f90 0x7fb9e80b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:47.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9ecc3c700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e809c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 -- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 msgr2=0x7fb9e809c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e809c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 -- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9e8073f90 msgr2=0x7fb9e80b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9e8073f90 0x7fb9e80b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 -- 172.21.15.77:0/256278604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0009a70 con 0x7fb9e80d0630 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9ecc3c700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e809c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e77fe700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 0x7fb9e81f4780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c490 tx=0x7fb9d800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8005cb0 con 0x7fb9e80d0630 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d8005e10 con 0x7fb9e80d0630 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc009550 con 0x7fb9e80d0630 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80072d0 con 0x7fb9e80d0630 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.947+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e809ab10 con 0x7fb9e80d0630 2024-01-15T14:36:47.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.948+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e8069020 con 0x7fb9e80d0630 2024-01-15T14:36:47.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.949+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fb9d8007430 con 0x7fb9e80d0630 2024-01-15T14:36:47.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.949+0000 7fb9e4ff9700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb9d406b500 0x7fb9d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:47.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.949+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9d8089170 con 0x7fb9e80d0630 2024-01-15T14:36:47.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.949+0000 7fb9e7fff700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb9d406b500 0x7fb9d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:47.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.949+0000 7fb9e7fff700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb9d406b500 0x7fb9d406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009c40 tx=0x7fb9dc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:47.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:47.953+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9d8053e10 con 0x7fb9e80d0630 2024-01-15T14:36:48.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.186+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9e806c830 con 0x7fb9d406b500 2024-01-15T14:36:48.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.913+0000 7fb9e4ff9700 1 -- 172.21.15.77:0/256278604 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fb9dc011b80 con 0x7fb9d406b500 2024-01-15T14:36:48.914 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-01-15T14:36:48.914 INFO:teuthology.orchestra.run.smithi077.stdout: "status": { 2024-01-15T14:36:48.914 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "complete" 2024-01-15T14:36:48.914 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:48.914 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.915+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb9d406b500 msgr2=0x7fb9d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.915+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb9d406b500 0x7fb9d406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009c40 tx=0x7fb9dc008030 comp rx=0 tx=0).stop 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.915+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 msgr2=0x7fb9e81f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.915+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 0x7fb9e81f4780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c490 tx=0x7fb9d800c7a0 comp rx=0 tx=0).stop 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 shutdown_connections 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fb9d406b500 0x7fb9d406d9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fb9e815c030 0x7fb9e809c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fb9e8073f90 0x7fb9e80b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 --2- 172.21.15.77:0/256278604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80d0630 0x7fb9e81f4780 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 >> 172.21.15.77:0/256278604 conn(0x7fb9e80d0c00 msgr2=0x7fb9e815e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 shutdown_connections 2024-01-15T14:36:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:48.916+0000 7fb9ee69f700 1 -- 172.21.15.77:0/256278604 wait complete. 2024-01-15T14:36:48.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T14:36:49.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 Processor -- start 2024-01-15T14:36:49.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 -- start start 2024-01-15T14:36:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb801473a0 0x7fcb8015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb80147780 0x7fcb8006c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8006c9a0 con 0x7fcb8006dc20 2024-01-15T14:36:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8015eb00 con 0x7fcb80147780 2024-01-15T14:36:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb8516f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8015ede0 con 0x7fcb801473a0 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.054+0000 7fcb7e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb80147780 0x7fcb8006c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb80147780 0x7fcb8006c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33498/0 (socket says 172.21.15.77:33498) 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 -- 172.21.15.77:0/341180184 learned_addr learned my addr 172.21.15.77:0/341180184 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 -- 172.21.15.77:0/341180184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb801473a0 msgr2=0x7fcb8015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 --2- 172.21.15.77:0/341180184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb801473a0 0x7fcb8015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 -- 172.21.15.77:0/341180184 >> v1:172.21.15.159:6789/0 conn(0x7fcb8006dc20 legacy=0x7fcb80073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 -- 172.21.15.77:0/341180184 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8015f0c0 con 0x7fcb80147780 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7e59c700 1 --2- 172.21.15.77:0/341180184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb80147780 0x7fcb8006c460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098f0 tx=0x7fcb74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68903af807e2796 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7dd9b700 1 -- 172.21.15.77:0/341180184 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74004230 con 0x7fcb80147780 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7dd9b700 1 -- 172.21.15.77:0/341180184 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74004390 con 0x7fcb80147780 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.055+0000 7fcb7dd9b700 1 -- 172.21.15.77:0/341180184 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740036a0 con 0x7fcb80147780 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 -- 172.21.15.77:0/341180184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb80147780 msgr2=0x7fcb8006c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 --2- 172.21.15.77:0/341180184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb80147780 0x7fcb8006c460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098f0 tx=0x7fcb74009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 -- 172.21.15.77:0/341180184 shutdown_connections 2024-01-15T14:36:49.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 --2- 172.21.15.77:0/341180184 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb801473a0 0x7fcb8015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 --2- 172.21.15.77:0/341180184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb80147780 0x7fcb8006c460 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 -- 172.21.15.77:0/341180184 >> 172.21.15.77:0/341180184 conn(0x7fcb800b3130 msgr2=0x7fcb800d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:49.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 -- 172.21.15.77:0/341180184 shutdown_connections 2024-01-15T14:36:49.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.056+0000 7fcb8516f700 1 -- 172.21.15.77:0/341180184 wait complete. 2024-01-15T14:36:49.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.057+0000 7fcb8516f700 1 Processor -- start 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.057+0000 7fcb8516f700 1 -- start start 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb8516f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 0x7fcb8015d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb8516f700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb801473a0 0x7fcb80161e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb8516f700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb80147780 0x7fcb80162360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb8516f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b1e80 con 0x7fcb8006dc20 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb8516f700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b1200 con 0x7fcb801473a0 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb8516f700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b14e0 con 0x7fcb80147780 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 0x7fcb8015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 0x7fcb8015d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33512/0 (socket says 172.21.15.77:33512) 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 -- 172.21.15.77:0/2270819705 learned_addr learned my addr 172.21.15.77:0/2270819705 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7e59c700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb801473a0 0x7fcb80161e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7f59e700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb80147780 0x7fcb80162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 -- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb80147780 msgr2=0x7fcb80162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb80147780 0x7fcb80162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 -- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb801473a0 msgr2=0x7fcb80161e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb801473a0 0x7fcb80161e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 -- 172.21.15.77:0/2270819705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb74009550 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7e59c700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb801473a0 0x7fcb80161e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7f59e700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb80147780 0x7fcb80162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.058+0000 7fcb7ed9d700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 0x7fcb8015d650 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000b530 tx=0x7fcb7000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.059+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70011830 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.059+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb700119b0 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.059+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb800b17c0 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.059+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70011d60 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.059+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8015db90 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.060+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb800b1fc0 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.060+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fcb7001a400 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.060+0000 7fcb677fe700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.060+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb70089ee0 con 0x7fcb8006dc20 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.060+0000 7fcb7e59c700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.061+0000 7fcb7e59c700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098f0 tx=0x7fcb74008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.065+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb70054b60 con 0x7fcb8006dc20 2024-01-15T14:36:49.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.285+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb8006b090 con 0x7fcb6c06b500 2024-01-15T14:36:49.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.286+0000 7fcb677fe700 1 -- 172.21.15.77:0/2270819705 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcb7401f070 con 0x7fcb6c06b500 2024-01-15T14:36:49.287 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-01-15T14:36:49.287 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:49.287 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephfs" 2024-01-15T14:36:49.287 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:49.287 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-01-15T14:36:49.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fcb6c06b500 msgr2=0x7fcb6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098f0 tx=0x7fcb74008030 comp rx=0 tx=0).stop 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 msgr2=0x7fcb8015d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 0x7fcb8015d650 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000b530 tx=0x7fcb7000b8f0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 shutdown_connections 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fcb80147780 0x7fcb80162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb8006dc20 0x7fcb8015d650 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 --2- 172.21.15.77:0/2270819705 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fcb801473a0 0x7fcb80161e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 >> 172.21.15.77:0/2270819705 conn(0x7fcb800b3130 msgr2=0x7fcb8015f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 shutdown_connections 2024-01-15T14:36:49.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.288+0000 7fcb8516f700 1 -- 172.21.15.77:0/2270819705 wait complete. 2024-01-15T14:36:49.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T14:36:49.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.424+0000 7f5cc7c4c700 1 Processor -- start 2024-01-15T14:36:49.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.425+0000 7f5cc7c4c700 1 -- start start 2024-01-15T14:36:49.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc7c4c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc006f580 0x7f5cc015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc7c4c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f5cc015ec50 0x7f5cc006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc7c4c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc00b5600 con 0x7f5cc0158600 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc7c4c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc015f0c0 con 0x7f5cc006f580 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc7c4c700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc006fa10 con 0x7f5cc015ec50 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc006f580 0x7f5cc015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc59e8700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5cc0158600 0x7f5cc00b4ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49904/0 (socket says 172.21.15.77:49904) 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc59e8700 1 -- 172.21.15.77:0/2000237845 learned_addr learned my addr 172.21.15.77:0/2000237845 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc006f580 0x7f5cc015e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36942/0 (socket says 172.21.15.77:36942) 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 -- 172.21.15.77:0/2000237845 >> v2:172.21.15.159:3300/0 conn(0x7f5cc015ec50 msgr2=0x7f5cc006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 --2- 172.21.15.77:0/2000237845 >> v2:172.21.15.159:3300/0 conn(0x7f5cc015ec50 0x7f5cc006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 -- 172.21.15.77:0/2000237845 >> v1:172.21.15.77:6789/0 conn(0x7f5cc0158600 legacy=0x7f5cc00b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 -- 172.21.15.77:0/2000237845 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc006e4e0 con 0x7f5cc006f580 2024-01-15T14:36:49.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc51e7700 1 --2- 172.21.15.77:0/2000237845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc006f580 0x7f5cc015e710 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009840 tx=0x7f5cbc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc27fec07aef870c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.426+0000 7f5cc49e6700 1 -- 172.21.15.77:0/2000237845 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00f8a0 con 0x7f5cc006f580 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc49e6700 1 -- 172.21.15.77:0/2000237845 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00fa00 con 0x7f5cc006f580 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc49e6700 1 -- 172.21.15.77:0/2000237845 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00fd90 con 0x7f5cc006f580 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/2000237845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc006f580 msgr2=0x7f5cc015e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/2000237845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc006f580 0x7f5cc015e710 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009840 tx=0x7f5cbc009b50 comp rx=0 tx=0).stop 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/2000237845 shutdown_connections 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/2000237845 >> v2:172.21.15.159:3300/0 conn(0x7f5cc015ec50 0x7f5cc006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/2000237845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc006f580 0x7f5cc015e710 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/2000237845 >> 172.21.15.77:0/2000237845 conn(0x7f5cc00b3250 msgr2=0x7f5cc00d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/2000237845 shutdown_connections 2024-01-15T14:36:49.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.427+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/2000237845 wait complete. 2024-01-15T14:36:49.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.429+0000 7f5cc7c4c700 1 Processor -- start 2024-01-15T14:36:49.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.429+0000 7f5cc7c4c700 1 -- start start 2024-01-15T14:36:49.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cc006f580 0x7f5cc014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5cc0158600 0x7f5cc0152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 0x7f5cc00cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00ce8f0 con 0x7f5cc015ec50 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc0154e30 con 0x7f5cc0158600 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00ceb90 con 0x7f5cc006f580 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 0x7f5cc00cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 0x7f5cc00cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36956/0 (socket says 172.21.15.77:36956) 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 -- 172.21.15.77:0/1630905224 learned_addr learned my addr 172.21.15.77:0/1630905224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc51e7700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5cc0158600 0x7f5cc0152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc59e8700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cc006f580 0x7f5cc014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 -- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cc006f580 msgr2=0x7f5cc014a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cc006f580 0x7f5cc014a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 -- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5cc0158600 msgr2=0x7f5cc0152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5cc0158600 0x7f5cc0152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 -- 172.21.15.77:0/1630905224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0094a0 con 0x7f5cc015ec50 2024-01-15T14:36:49.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc61e9700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 0x7f5cc00cb8c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b970 tx=0x7f5cb800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800c930 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb800cab0 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc00cee70 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800ce60 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.430+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc014aeb0 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.431+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc00cf390 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.432+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f5cb801d3b0 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.432+0000 7f5cb67fc700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ca406b500 0x7f5ca406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.432+0000 7f5cc59e8700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ca406b500 0x7f5ca406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.432+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cb8089bc0 con 0x7f5cc015ec50 2024-01-15T14:36:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.433+0000 7f5cc59e8700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ca406b500 0x7f5ca406d9c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0005910 tx=0x7f5cb0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.437+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cc00cf390 con 0x7f5cc015ec50 2024-01-15T14:36:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.658+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5cc006ed80 con 0x7f5ca406b500 2024-01-15T14:36:49.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.662+0000 7f5cb67fc700 1 -- 172.21.15.77:0/1630905224 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f5cb0005a10 con 0x7f5ca406b500 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sv1" 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ss1clone1" 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:49.663 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.663+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ca406b500 msgr2=0x7f5ca406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.663+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ca406b500 0x7f5ca406d9c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0005910 tx=0x7f5cb0009040 comp rx=0 tx=0).stop 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.663+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 msgr2=0x7f5cc00cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 0x7f5cc00cb8c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b970 tx=0x7f5cb800bc80 comp rx=0 tx=0).stop 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 shutdown_connections 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f5ca406b500 0x7f5ca406d9c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f5cc006f580 0x7f5cc014a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc015ec50 0x7f5cc00cb8c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 --2- 172.21.15.77:0/1630905224 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f5cc0158600 0x7f5cc0152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 >> 172.21.15.77:0/1630905224 conn(0x7f5cc00b3250 msgr2=0x7f5cc0073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:49.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 shutdown_connections 2024-01-15T14:36:49.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.664+0000 7f5cc7c4c700 1 -- 172.21.15.77:0/1630905224 wait complete. 2024-01-15T14:36:49.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T14:36:49.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.804+0000 7f7fdffb7700 1 Processor -- start 2024-01-15T14:36:49.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.804+0000 7f7fdffb7700 1 -- start start 2024-01-15T14:36:49.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdffb7700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 0x7f7fd806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdffb7700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd806bed0 con 0x7f7fd8162820 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdffb7700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd806bba0 con 0x7f7fd81622b0 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdffb7700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8070060 con 0x7f7fd815df20 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdd552700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7fd8162820 0x7f7fd8161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49910/0 (socket says 172.21.15.77:49910) 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdd552700 1 -- 172.21.15.77:0/4211727801 learned_addr learned my addr 172.21.15.77:0/4211727801 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195259285 0 0) 0x7f7fd806bed0 con 0x7f7fd8162820 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fddd53700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 0x7f7fd806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc40035b0 con 0x7f7fd8162820 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1230566415 0 0) 0x7f7fc40035b0 con 0x7f7fd8162820 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 msgr2=0x7f7fd806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 0x7f7fd806c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 >> v1:172.21.15.159:6789/0 conn(0x7f7fd81622b0 legacy=0x7f7fd80cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd806faa0 con 0x7f7fd8162820 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.805+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f7fd4001670 con 0x7f7fd8162820 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 >> v1:172.21.15.77:6789/0 conn(0x7f7fd8162820 legacy=0x7f7fd8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fc4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc4004850 0x7f7fc4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 0x7f7fc4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8070060 con 0x7f7fc4003a30 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc400bac0 con 0x7f7fc4004850 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc400bc80 con 0x7f7fd815df20 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fc4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdd552700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc4004850 0x7f7fc4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fde554700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 0x7f7fc4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 -- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 msgr2=0x7f7fc4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd815df20 0x7f7fc4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 -- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc4004850 msgr2=0x7f7fc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fc4004850 0x7f7fc4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 -- 172.21.15.77:0/4211727801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc40035b0 con 0x7f7fc4003a30 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fddd53700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fc4003e50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009b30 tx=0x7f7fc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41c6d0d9c083457 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc800e6c0 con 0x7f7fc4003a30 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc800e820 con 0x7f7fc4003a30 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.806+0000 7f7fdcd51700 1 -- 172.21.15.77:0/4211727801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc800eb90 con 0x7f7fc4003a30 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fdffb7700 1 -- 172.21.15.77:0/4211727801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 msgr2=0x7f7fc4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/4211727801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fc4003e50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009b30 tx=0x7f7fc4005880 comp rx=0 tx=0).stop 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fde554700 1 -- 172.21.15.77:0/4211727801 reap_dead start 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fdffb7700 1 -- 172.21.15.77:0/4211727801 shutdown_connections 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fdffb7700 1 -- 172.21.15.77:0/4211727801 >> 172.21.15.77:0/4211727801 conn(0x7f7fd806dc80 msgr2=0x7f7fd809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:49.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fdffb7700 1 -- 172.21.15.77:0/4211727801 shutdown_connections 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.807+0000 7f7fdffb7700 1 -- 172.21.15.77:0/4211727801 wait complete. 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.808+0000 7f7fdffb7700 1 Processor -- start 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.808+0000 7f7fdffb7700 1 -- start start 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fd8156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fd81571d0 0x7f7fd80c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd80c8ee0 0x7f7fd81e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80c97e0 con 0x7f7fc4003a30 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80c9ac0 con 0x7f7fd81571d0 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80c9da0 con 0x7f7fd80c8ee0 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fd8156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fd8156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36964/0 (socket says 172.21.15.77:36964) 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 -- 172.21.15.77:0/59144298 learned_addr learned my addr 172.21.15.77:0/59144298 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdd552700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fd81571d0 0x7f7fd80c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fde554700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd80c8ee0 0x7f7fd81e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 -- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd80c8ee0 msgr2=0x7f7fd81e79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd80c8ee0 0x7f7fd81e79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 -- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fd81571d0 msgr2=0x7f7fd80c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fd81571d0 0x7f7fd80c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 -- 172.21.15.77:0/59144298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd80ca080 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fddd53700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fd8156b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40047b0 tx=0x7f7fc80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc800e6c0 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc8003ad0 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd80ca360 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8003e80 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.809+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd80c9540 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.810+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd8067f60 con 0x7f7fc4003a30 2024-01-15T14:36:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.811+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f7fc8013470 con 0x7f7fc4003a30 2024-01-15T14:36:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.811+0000 7f7fce7fc700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7fbc06b500 0x7f7fbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.811+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fc808b780 con 0x7f7fc4003a30 2024-01-15T14:36:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.811+0000 7f7fdd552700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7fbc06b500 0x7f7fbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.811+0000 7f7fdd552700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7fbc06b500 0x7f7fbc06d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4001eb0 tx=0x7f7fd4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:49.816+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fc8056520 con 0x7f7fc4003a30 2024-01-15T14:36:50.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.042+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fd80c9c30 con 0x7f7fbc06b500 2024-01-15T14:36:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.053+0000 7f7fce7fc700 1 -- 172.21.15.77:0/59144298 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f7fd80c9c30 con 0x7f7fbc06b500 2024-01-15T14:36:50.054 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:50.054 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-01-15T14:36:50.055 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-01-15T14:36:50.055 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ss1" 2024-01-15T14:36:50.055 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-01-15T14:36:50.055 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-01-15T14:36:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7fbc06b500 msgr2=0x7f7fbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7fbc06b500 0x7f7fbc06d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4001eb0 tx=0x7f7fd4009040 comp rx=0 tx=0).stop 2024-01-15T14:36:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 msgr2=0x7f7fd8156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fd8156b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40047b0 tx=0x7f7fc80047c0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 shutdown_connections 2024-01-15T14:36:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f7fbc06b500 0x7f7fbc06d9c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f7fd80c8ee0 0x7f7fd81e79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fc4003a30 0x7f7fd8156b40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 --2- 172.21.15.77:0/59144298 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f7fd81571d0 0x7f7fd80c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 >> 172.21.15.77:0/59144298 conn(0x7f7fd806dc80 msgr2=0x7f7fd80ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 shutdown_connections 2024-01-15T14:36:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.056+0000 7f7fdffb7700 1 -- 172.21.15.77:0/59144298 wait complete. 2024-01-15T14:36:50.069 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T14:36:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.196+0000 7f1aa778c700 1 Processor -- start 2024-01-15T14:36:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.196+0000 7f1aa778c700 1 -- start start 2024-01-15T14:36:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa778c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1aa01555d0 0x7f1aa00b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa778c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f1aa00b40d0 0x7f1aa006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa778c700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa006d440 con 0x7f1aa00713e0 2024-01-15T14:36:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa778c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa006da70 con 0x7f1aa01555d0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa778c700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00b4900 con 0x7f1aa00b40d0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa4d27700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1aa01555d0 0x7f1aa00b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa4d27700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1aa01555d0 0x7f1aa00b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36974/0 (socket says 172.21.15.77:36974) 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.197+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2844431609 learned_addr learned my addr 172.21.15.77:0/2844431609 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1aa5d29700 1 --2- 172.21.15.77:0/2844431609 >> v2:172.21.15.159:3300/0 conn(0x7f1aa00b40d0 0x7f1aa006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2844431609 >> v2:172.21.15.159:3300/0 conn(0x7f1aa00b40d0 msgr2=0x7f1aa006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1aa4d27700 1 --2- 172.21.15.77:0/2844431609 >> v2:172.21.15.159:3300/0 conn(0x7f1aa00b40d0 0x7f1aa006de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2844431609 >> v1:172.21.15.159:6789/0 conn(0x7f1aa00713e0 legacy=0x7f1aa00717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2844431609 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00b4be0 con 0x7f1aa01555d0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1aa4d27700 1 --2- 172.21.15.77:0/2844431609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa01555d0 0x7f1aa00b3b90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00624d0 tx=0x7f1a9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d05ff19f5477188 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1a97fff700 1 -- 172.21.15.77:0/2844431609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c004230 con 0x7f1aa01555d0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1a97fff700 1 -- 172.21.15.77:0/2844431609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c004390 con 0x7f1aa01555d0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.198+0000 7f1a97fff700 1 -- 172.21.15.77:0/2844431609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c0036a0 con 0x7f1aa01555d0 2024-01-15T14:36:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 -- 172.21.15.77:0/2844431609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa01555d0 msgr2=0x7f1aa00b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2844431609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa01555d0 0x7f1aa00b3b90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00624d0 tx=0x7f1a9c009bd0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 -- 172.21.15.77:0/2844431609 shutdown_connections 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2844431609 >> v2:172.21.15.159:3300/0 conn(0x7f1aa00b40d0 0x7f1aa006de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2844431609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa01555d0 0x7f1aa00b3b90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 -- 172.21.15.77:0/2844431609 >> 172.21.15.77:0/2844431609 conn(0x7f1aa00b20c0 msgr2=0x7f1aa00b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 -- 172.21.15.77:0/2844431609 shutdown_connections 2024-01-15T14:36:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.199+0000 7f1aa778c700 1 -- 172.21.15.77:0/2844431609 wait complete. 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.200+0000 7f1aa778c700 1 Processor -- start 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.200+0000 7f1aa778c700 1 -- start start 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1aa00713e0 0x7f1aa0150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 0x7f1aa014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1aa01555d0 0x7f1aa014add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa01367e0 con 0x7f1aa00b40d0 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0150d20 con 0x7f1aa01555d0 2024-01-15T14:36:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0151000 con 0x7f1aa00713e0 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 0x7f1aa014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 0x7f1aa014a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36978/0 (socket says 172.21.15.77:36978) 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2819469739 learned_addr learned my addr 172.21.15.77:0/2819469739 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa5528700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1aa00713e0 0x7f1aa0150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa5d29700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1aa01555d0 0x7f1aa014add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1aa00713e0 msgr2=0x7f1aa0150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1aa00713e0 0x7f1aa0150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1aa01555d0 msgr2=0x7f1aa014add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1aa01555d0 0x7f1aa014add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 -- 172.21.15.77:0/2819469739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c009550 con 0x7f1aa00b40d0 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa5528700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1aa00713e0 0x7f1aa0150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa5d29700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1aa01555d0 0x7f1aa014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa4d27700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 0x7f1aa014a890 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009130 tx=0x7f1a9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c011630 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c011790 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa01512e0 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.201+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c011b40 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.202+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa01518a0 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.202+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa0069020 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.203+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f1a9c011ca0 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.203+0000 7f1a95ffb700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1a8406b500 0x7f1a8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.203+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a9c08b450 con 0x7f1aa00b40d0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.203+0000 7f1aa5528700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1a8406b500 0x7f1a8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.204+0000 7f1aa5528700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1a8406b500 0x7f1a8406d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a90003fc0 tx=0x7f1a90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.208+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a9c0562b0 con 0x7f1aa00b40d0 2024-01-15T14:36:50.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.442+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1aa014c700 con 0x7f1a8406b500 2024-01-15T14:36:50.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.466+0000 7f1a95ffb700 1 -- 172.21.15.77:0/2819469739 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1aa014c700 con 0x7f1a8406b500 2024-01-15T14:36:50.467 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:50.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1a8406b500 msgr2=0x7f1a8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1a8406b500 0x7f1a8406d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a90003fc0 tx=0x7f1a90009040 comp rx=0 tx=0).stop 2024-01-15T14:36:50.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 msgr2=0x7f1aa014a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 0x7f1aa014a890 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009130 tx=0x7f1a9c004750 comp rx=0 tx=0).stop 2024-01-15T14:36:50.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 shutdown_connections 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f1a8406b500 0x7f1a8406d9c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f1aa00713e0 0x7f1aa0150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f1aa01555d0 0x7f1aa014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 --2- 172.21.15.77:0/2819469739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa00b40d0 0x7f1aa014a890 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 >> 172.21.15.77:0/2819469739 conn(0x7f1aa00b20c0 msgr2=0x7f1aa006e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 shutdown_connections 2024-01-15T14:36:50.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.469+0000 7f1aa778c700 1 -- 172.21.15.77:0/2819469739 wait complete. 2024-01-15T14:36:50.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T14:36:50.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.606+0000 7f2f5e791700 1 Processor -- start 2024-01-15T14:36:50.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.606+0000 7f2f5e791700 1 -- start start 2024-01-15T14:36:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f5e791700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f2f5815bcd0 0x7f2f58071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f5e791700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 0x7f2f5806b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f5e791700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f58072500 con 0x7f2f581554d0 2024-01-15T14:36:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f5e791700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f580ceb80 con 0x7f2f5815bcd0 2024-01-15T14:36:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f5e791700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580d05b0 con 0x7f2f580720d0 2024-01-15T14:36:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f57fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2f581554d0 0x7f2f5806d910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49912/0 (socket says 172.21.15.77:49912) 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f57fff700 1 -- 172.21.15.77:0/1973203283 learned_addr learned my addr 172.21.15.77:0/1973203283 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368735922 0 0) 0x7f2f58072500 con 0x7f2f581554d0 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f577fe700 1 --2- 172.21.15.77:0/1973203283 >> v2:172.21.15.159:3300/0 conn(0x7f2f5815bcd0 0x7f2f58071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f5cd2e700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 0x7f2f5806b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f400035b0 con 0x7f2f581554d0 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545305133 0 0) 0x7f2f400035b0 con 0x7f2f581554d0 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 msgr2=0x7f2f5806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 0x7f2f5806b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 >> v2:172.21.15.159:3300/0 conn(0x7f2f5815bcd0 msgr2=0x7f2f58071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 --2- 172.21.15.77:0/1973203283 >> v2:172.21.15.159:3300/0 conn(0x7f2f5815bcd0 0x7f2f58071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580d0890 con 0x7f2f581554d0 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.607+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f2f4c001670 con 0x7f2f581554d0 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 >> v1:172.21.15.77:6789/0 conn(0x7f2f581554d0 legacy=0x7f2f5806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f40003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 0x7f2f40004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f40004d90 0x7f2f40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580ceb80 con 0x7f2f40003a00 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580d05b0 con 0x7f2f40004d90 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000ba40 con 0x7f2f580720d0 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f40003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f5cd2e700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f40004d90 0x7f2f40005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f577fe700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 0x7f2f40004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 -- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 msgr2=0x7f2f40004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f580720d0 0x7f2f40004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 -- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f40004d90 msgr2=0x7f2f40005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f40004d90 0x7f2f40005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 -- 172.21.15.77:0/1973203283 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f400035b0 con 0x7f2f40003a00 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f5cd2e700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f40004d90 0x7f2f40005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f57fff700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f40003e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00c320 tx=0x7f2f40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7847343b34514f52 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c00ce60 con 0x7f2f40003a00 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c005960 con 0x7f2f40003a00 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.608+0000 7f2f56ffd700 1 -- 172.21.15.77:0/1973203283 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c005cf0 con 0x7f2f40003a00 2024-01-15T14:36:50.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5e791700 1 -- 172.21.15.77:0/1973203283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 msgr2=0x7f2f40003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5e791700 1 --2- 172.21.15.77:0/1973203283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f40003e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00c320 tx=0x7f2f40005880 comp rx=0 tx=0).stop 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5cd2e700 1 -- 172.21.15.77:0/1973203283 reap_dead start 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5e791700 1 -- 172.21.15.77:0/1973203283 shutdown_connections 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5e791700 1 -- 172.21.15.77:0/1973203283 >> 172.21.15.77:0/1973203283 conn(0x7f2f580cd6b0 msgr2=0x7f2f580cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5e791700 1 -- 172.21.15.77:0/1973203283 shutdown_connections 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.609+0000 7f2f5e791700 1 -- 172.21.15.77:0/1973203283 wait complete. 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.610+0000 7f2f5e791700 1 Processor -- start 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.610+0000 7f2f5e791700 1 -- start start 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5e791700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f581529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5e791700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f581f48c0 0x7f2f5806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5e791700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f5806ece0 0x7f2f58159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5e791700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580b06e0 con 0x7f2f40003a00 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5e791700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580b0930 con 0x7f2f581f48c0 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5e791700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580b0c10 con 0x7f2f5806ece0 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f581529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f581529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37002/0 (socket says 172.21.15.77:37002) 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 -- 172.21.15.77:0/3173532885 learned_addr learned my addr 172.21.15.77:0/3173532885 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f5cd2e700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f5806ece0 0x7f2f58159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f577fe700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f581f48c0 0x7f2f5806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 -- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f5806ece0 msgr2=0x7f2f58159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f5806ece0 0x7f2f58159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 -- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f581f48c0 msgr2=0x7f2f5806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f581f48c0 0x7f2f5806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 -- 172.21.15.77:0/3173532885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580b0ef0 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.611+0000 7f2f57fff700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f581529f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c008fd0 tx=0x7f2f4c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.612+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c004870 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.612+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c0049d0 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.612+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f580b11d0 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.612+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c004d80 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.612+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f581f4d40 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.613+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f581f4f30 con 0x7f2f40003a00 2024-01-15T14:36:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.613+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f2f4c0070d0 con 0x7f2f40003a00 2024-01-15T14:36:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.614+0000 7f2f54ff9700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2f4406b500 0x7f2f4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.614+0000 7f2f577fe700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2f4406b500 0x7f2f4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.614+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f4c08b530 con 0x7f2f40003a00 2024-01-15T14:36:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.614+0000 7f2f577fe700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2f4406b500 0x7f2f4406d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2f5815a8e0 tx=0x7f2f48008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.618+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f4c056310 con 0x7f2f40003a00 2024-01-15T14:36:50.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.840+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f5815c160 con 0x7f2f4406b500 2024-01-15T14:36:50.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.855+0000 7f2f54ff9700 1 -- 172.21.15.77:0/3173532885 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2f48004030 con 0x7f2f4406b500 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2f4406b500 msgr2=0x7f2f4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2f4406b500 0x7f2f4406d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2f5815a8e0 tx=0x7f2f48008040 comp rx=0 tx=0).stop 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 msgr2=0x7f2f581529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f581529f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c008fd0 tx=0x7f2f4c001a00 comp rx=0 tx=0).stop 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 shutdown_connections 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f2f4406b500 0x7f2f4406d9c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f2f5806ece0 0x7f2f58159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f40003a00 0x7f2f581529f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 --2- 172.21.15.77:0/3173532885 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f2f581f48c0 0x7f2f5806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 >> 172.21.15.77:0/3173532885 conn(0x7f2f580cd6b0 msgr2=0x7f2f5806bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 shutdown_connections 2024-01-15T14:36:50.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:50.857+0000 7f2f5e791700 1 -- 172.21.15.77:0/3173532885 wait complete. 2024-01-15T14:36:50.860 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.858+0000 7feba0223700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.860 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.858+0000 7feba0223700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.860 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.858+0000 7feba0223700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.860 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.858+0000 7feba0223700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.860 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.858+0000 7feba0223700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T14:36:50.877 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.876+0000 7feb9ea20700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.877 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.876+0000 7feb9ea20700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.877 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.876+0000 7feb9ea20700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.877 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.876+0000 7feb9ea20700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:50.878 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-01-15T14:36:50.876+0000 7feb9ea20700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T14:36:51.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.003+0000 7fdfe4cd3700 1 Processor -- start 2024-01-15T14:36:51.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.003+0000 7fdfe4cd3700 1 -- start start 2024-01-15T14:36:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfe4cd3700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdfe00b1290 0x7fdfe00d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfe4cd3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cc1f0 0x7fdfe00d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfe4cd3700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe00b1cb0 con 0x7fdfe00cfef0 2024-01-15T14:36:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfe4cd3700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe00b1670 con 0x7fdfe00b1290 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfe4cd3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00d0c60 con 0x7fdfe00cc1f0 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfde59c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdfe00b1290 0x7fdfe00d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfddd9b700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cc1f0 0x7fdfe00d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfded9d700 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fdfe00cfef0 0x7fdfe00d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.77:41794/0 (socket says 172.21.15.77:41794) 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfde59c700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7fdfe00b1290 0x7fdfe00d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.159:3300/0 says I am v2:172.21.15.77:57648/0 (socket says 172.21.15.77:57648) 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.004+0000 7fdfded9d700 1 -- 172.21.15.77:0/254575396 learned_addr learned my addr 172.21.15.77:0/254575396 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfde59c700 1 -- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cc1f0 msgr2=0x7fdfe00d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfde59c700 1 --2- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cc1f0 0x7fdfe00d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfde59c700 1 -- 172.21.15.77:0/254575396 >> v1:172.21.15.159:6789/0 conn(0x7fdfe00cfef0 legacy=0x7fdfe00d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfde59c700 1 -- 172.21.15.77:0/254575396 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe00d0f40 con 0x7fdfe00b1290 2024-01-15T14:36:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfdd59a700 1 -- 172.21.15.77:0/254575396 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396557073 0 0) 0x7fdfe00b1cb0 con 0x7fdfe00cfef0 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfddd9b700 1 --2- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cc1f0 0x7fdfe00d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfde59c700 1 --2- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00b1290 0x7fdfe00d1850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009840 tx=0x7fdfd0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e935c10beb5b0ee server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfdd59a700 1 -- 172.21.15.77:0/254575396 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0011830 con 0x7fdfe00b1290 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfdd59a700 1 -- 172.21.15.77:0/254575396 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd0011990 con 0x7fdfe00b1290 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.005+0000 7fdfdd59a700 1 -- 172.21.15.77:0/254575396 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0011d20 con 0x7fdfe00b1290 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00b1290 msgr2=0x7fdfe00d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00b1290 0x7fdfe00d1850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009840 tx=0x7fdfd0009b50 comp rx=0 tx=0).stop 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/254575396 shutdown_connections 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00b1290 0x7fdfe00d1850 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/254575396 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cc1f0 0x7fdfe00d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/254575396 >> 172.21.15.77:0/254575396 conn(0x7fdfe009aa00 msgr2=0x7fdfe009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/254575396 shutdown_connections 2024-01-15T14:36:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.006+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/254575396 wait complete. 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.007+0000 7fdfe4cd3700 1 Processor -- start 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.007+0000 7fdfe4cd3700 1 -- start start 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfe4cd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 0x7fdfe00b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfe4cd3700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00cc1f0 0x7fdfe01520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfe4cd3700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cfef0 0x7fdfe00b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfe4cd3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00b4d40 con 0x7fdfe00b1290 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfe4cd3700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00b5020 con 0x7fdfe00cc1f0 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfe4cd3700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00b5300 con 0x7fdfe00cfef0 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 0x7fdfe00b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 0x7fdfe00b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37044/0 (socket says 172.21.15.77:37044) 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 -- 172.21.15.77:0/1253864690 learned_addr learned my addr 172.21.15.77:0/1253864690 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfddd9b700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00cc1f0 0x7fdfe01520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfded9d700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cfef0 0x7fdfe00b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 -- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cfef0 msgr2=0x7fdfe00b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cfef0 0x7fdfe00b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 -- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00cc1f0 msgr2=0x7fdfe01520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00cc1f0 0x7fdfe01520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 -- 172.21.15.77:0/1253864690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40094a0 con 0x7fdfe00b1290 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.008+0000 7fdfde59c700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 0x7fdfe00b5de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009b90 tx=0x7fdfd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.009+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00038a0 con 0x7fdfe00b1290 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.009+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd001f070 con 0x7fdfe00b1290 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.009+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd00094a0 con 0x7fdfe00b1290 2024-01-15T14:36:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.009+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0003c20 con 0x7fdfe00b1290 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.009+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe00b58c0 con 0x7fdfe00b1290 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.010+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe0067f60 con 0x7fdfe00b1290 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.010+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fdfd001a430 con 0x7fdfe00b1290 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.011+0000 7fdfceffd700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fdfc006b530 0x7fdfc006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.011+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfd0021070 con 0x7fdfe00b1290 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.011+0000 7fdfddd9b700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fdfc006b530 0x7fdfc006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.011+0000 7fdfddd9b700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fdfc006b530 0x7fdfc006d9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00b68e0 tx=0x7fdfd4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:51.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.015+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfd0056880 con 0x7fdfe00b1290 2024-01-15T14:36:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.244+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe00b5190 con 0x7fdfc006b530 2024-01-15T14:36:51.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.261+0000 7fdfceffd700 1 -- 172.21.15.77:0/1253864690 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdfe00b5190 con 0x7fdfc006b530 2024-01-15T14:36:51.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fdfc006b530 msgr2=0x7fdfc006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fdfc006b530 0x7fdfc006d9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00b68e0 tx=0x7fdfd4008040 comp rx=0 tx=0).stop 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 msgr2=0x7fdfe00b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 0x7fdfe00b5de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009b90 tx=0x7fdfd0004750 comp rx=0 tx=0).stop 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 shutdown_connections 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fdfc006b530 0x7fdfc006d9f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fdfe00cfef0 0x7fdfe00b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfe00b1290 0x7fdfe00b5de0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 --2- 172.21.15.77:0/1253864690 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fdfe00cc1f0 0x7fdfe01520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 >> 172.21.15.77:0/1253864690 conn(0x7fdfe009aa00 msgr2=0x7fdfe0161170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 shutdown_connections 2024-01-15T14:36:51.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.263+0000 7fdfe4cd3700 1 -- 172.21.15.77:0/1253864690 wait complete. 2024-01-15T14:36:51.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:51.279 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T14:36:51.353 INFO:teuthology.orchestra.run.smithi077.stdout:2 2024-01-15T14:36:51.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.519+0000 7fe8a5dea700 1 Processor -- start 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.519+0000 7fe8a5dea700 1 -- start start 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe8a5dea700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe8a5dea700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8a00744d0 0x7fe8a00add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe8a5dea700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a015e4a0 con 0x7fe8a015e0c0 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe8a5dea700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a0074960 con 0x7fe8a00744d0 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe8a5dea700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a00affe0 con 0x7fe8a00afb80 2024-01-15T14:36:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8a00744d0 0x7fe8a00add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89f7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe8a015e0c0 0x7fe8a00b3bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49924/0 (socket says 172.21.15.77:49924) 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89f7fe700 1 -- 172.21.15.77:0/4255748002 learned_addr learned my addr 172.21.15.77:0/4255748002 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89effd700 1 --2- 172.21.15.77:0/4255748002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89ffff700 1 -- 172.21.15.77:0/4255748002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 msgr2=0x7fe8a0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89ffff700 1 --2- 172.21.15.77:0/4255748002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a0073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89ffff700 1 -- 172.21.15.77:0/4255748002 >> v1:172.21.15.77:6789/0 conn(0x7fe8a015e0c0 legacy=0x7fe8a00b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89e7fc700 1 -- 172.21.15.77:0/4255748002 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700689068 0 0) 0x7fe8a015e4a0 con 0x7fe8a015e0c0 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89ffff700 1 -- 172.21.15.77:0/4255748002 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a006aff0 con 0x7fe8a00744d0 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.520+0000 7fe89ffff700 1 --2- 172.21.15.77:0/4255748002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a00744d0 0x7fe8a00add40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe898009d90 tx=0x7fe89800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3c56403598465 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe89e7fc700 1 -- 172.21.15.77:0/4255748002 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89800e070 con 0x7fe8a00744d0 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe89e7fc700 1 -- 172.21.15.77:0/4255748002 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89800bbd0 con 0x7fe8a00744d0 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe89e7fc700 1 -- 172.21.15.77:0/4255748002 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898012430 con 0x7fe8a00744d0 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 -- 172.21.15.77:0/4255748002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a00744d0 msgr2=0x7fe8a00add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/4255748002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a00744d0 0x7fe8a00add40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe898009d90 tx=0x7fe89800b330 comp rx=0 tx=0).stop 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 -- 172.21.15.77:0/4255748002 shutdown_connections 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/4255748002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a00744d0 0x7fe8a00add40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/4255748002 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 -- 172.21.15.77:0/4255748002 >> 172.21.15.77:0/4255748002 conn(0x7fe8a00d0c00 msgr2=0x7fe8a00d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 -- 172.21.15.77:0/4255748002 shutdown_connections 2024-01-15T14:36:51.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.521+0000 7fe8a5dea700 1 -- 172.21.15.77:0/4255748002 wait complete. 2024-01-15T14:36:51.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.522+0000 7fe8a5dea700 1 Processor -- start 2024-01-15T14:36:51.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 -- start start 2024-01-15T14:36:51.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8a00744d0 0x7fe8a015e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a015ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 0x7fe8a015c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a0161210 con 0x7fe8a015e0c0 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a01608e0 con 0x7fe8a00744d0 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe8a5dea700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a0160bc0 con 0x7fe8a00afb80 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe89ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 0x7fe8a015c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe89ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 0x7fe8a015c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37058/0 (socket says 172.21.15.77:37058) 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.523+0000 7fe89ffff700 1 -- 172.21.15.77:0/2597719973 learned_addr learned my addr 172.21.15.77:0/2597719973 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89f7fe700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8a00744d0 0x7fe8a015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89effd700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a015ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89ffff700 1 -- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 msgr2=0x7fe8a015ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89ffff700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a015ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89ffff700 1 -- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8a00744d0 msgr2=0x7fe8a015e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89ffff700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8a00744d0 0x7fe8a015e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89ffff700 1 -- 172.21.15.77:0/2597719973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898009a70 con 0x7fe8a015e0c0 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89effd700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a015ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe89ffff700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 0x7fe8a015c700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe89800b370 tx=0x7fe898004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898012430 con 0x7fe8a015e0c0 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898020070 con 0x7fe8a015e0c0 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a0160ea0 con 0x7fe8a015e0c0 2024-01-15T14:36:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898004280 con 0x7fe8a015e0c0 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.524+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a0160500 con 0x7fe8a015e0c0 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.525+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a01613c0 con 0x7fe8a015e0c0 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.526+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fe8980043e0 con 0x7fe8a015e0c0 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.526+0000 7fe897fff700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe88006b530 0x7fe88006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.526+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe898022070 con 0x7fe8a015e0c0 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.526+0000 7fe89f7fe700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe88006b530 0x7fe88006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:51.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.526+0000 7fe89f7fe700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe88006b530 0x7fe88006d9f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe888007fd0 tx=0x7fe888007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:51.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.530+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8a01613c0 con 0x7fe8a015e0c0 2024-01-15T14:36:51.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:51.749+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 --> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8a006b890 con 0x7fe88006b530 2024-01-15T14:36:51.788 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-01-15T14:36:51.786+0000 7fe68c60c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T14:36:51.848 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-01-15T14:36:52.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:52.202+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 8) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fe898056de0 con 0x7fe8a015e0c0 2024-01-15T14:36:54.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.213+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe898059bc0 con 0x7fe8a015e0c0 2024-01-15T14:36:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.813+0000 7fe897fff700 1 -- 172.21.15.77:0/2597719973 <== mgr.4101 v2:172.21.15.77:6832/4256185546 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fe888001370 con 0x7fe88006b530 2024-01-15T14:36:54.814 INFO:teuthology.orchestra.run.smithi077.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T14:36:54.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.818+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe88006b530 msgr2=0x7fe88006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:54.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.818+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe88006b530 0x7fe88006d9f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe888007fd0 tx=0x7fe888007de0 comp rx=0 tx=0).stop 2024-01-15T14:36:54.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.818+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 msgr2=0x7fe8a015c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:54.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.818+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 0x7fe8a015c700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe89800b370 tx=0x7fe898004ad0 comp rx=0 tx=0).stop 2024-01-15T14:36:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.819+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 shutdown_connections 2024-01-15T14:36:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.819+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7fe88006b530 0x7fe88006d9f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.819+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8a015e0c0 0x7fe8a015c700 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.819+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7fe8a00744d0 0x7fe8a015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.819+0000 7fe8a5dea700 1 --2- 172.21.15.77:0/2597719973 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7fe8a00afb80 0x7fe8a015ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.819+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 >> 172.21.15.77:0/2597719973 conn(0x7fe8a00d0c00 msgr2=0x7fe8a00b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.820+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 shutdown_connections 2024-01-15T14:36:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:54.820+0000 7fe8a5dea700 1 -- 172.21.15.77:0/2597719973 wait complete. 2024-01-15T14:36:54.848 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T14:36:54.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T14:36:54.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T14:36:55.013 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T14:36:55.013 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:36:55.013 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:55.075 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-15T14:36:55.075 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T14:36:55.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T14:36:55.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.296+0000 7f9168fd4700 1 Processor -- start 2024-01-15T14:36:55.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.296+0000 7f9168fd4700 1 -- start start 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9168fd4700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9168fd4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91640ce090 0x7f916406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9168fd4700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916406e7f0 con 0x7f9164073f90 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9168fd4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916406dec0 con 0x7f91640ce090 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9168fd4700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916406e1a0 con 0x7f916415e2a0 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9161d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91640ce090 0x7f916406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9161d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91640ce090 0x7f916406ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37078/0 (socket says 172.21.15.77:37078) 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f9161d9b700 1 -- 172.21.15.77:0/2661526399 learned_addr learned my addr 172.21.15.77:0/2661526399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.297+0000 7f916259c700 1 --2- 172.21.15.77:0/2661526399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9161d9b700 1 -- 172.21.15.77:0/2661526399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 msgr2=0x7f916406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9161d9b700 1 --2- 172.21.15.77:0/2661526399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916406ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9161d9b700 1 -- 172.21.15.77:0/2661526399 >> v1:172.21.15.159:6789/0 conn(0x7f9164073f90 legacy=0x7f91640cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9161d9b700 1 -- 172.21.15.77:0/2661526399 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916406e480 con 0x7f91640ce090 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9161d9b700 1 --2- 172.21.15.77:0/2661526399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91640ce090 0x7f916406ec80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f91540098f0 tx=0x7f9154009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e7a703416dc907 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f916159a700 1 -- 172.21.15.77:0/2661526399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154011830 con 0x7f91640ce090 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f916159a700 1 -- 172.21.15.77:0/2661526399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9154011990 con 0x7f91640ce090 2024-01-15T14:36:55.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f916159a700 1 -- 172.21.15.77:0/2661526399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154011d20 con 0x7f91640ce090 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9168fd4700 1 -- 172.21.15.77:0/2661526399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91640ce090 msgr2=0x7f916406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.298+0000 7f9168fd4700 1 --2- 172.21.15.77:0/2661526399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91640ce090 0x7f916406ec80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f91540098f0 tx=0x7f9154009c00 comp rx=0 tx=0).stop 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.299+0000 7f9168fd4700 1 -- 172.21.15.77:0/2661526399 shutdown_connections 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.299+0000 7f9168fd4700 1 --2- 172.21.15.77:0/2661526399 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.299+0000 7f9168fd4700 1 --2- 172.21.15.77:0/2661526399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91640ce090 0x7f916406ec80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.299+0000 7f9168fd4700 1 -- 172.21.15.77:0/2661526399 >> 172.21.15.77:0/2661526399 conn(0x7f91640cd570 msgr2=0x7f91640cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:55.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.299+0000 7f9168fd4700 1 -- 172.21.15.77:0/2661526399 shutdown_connections 2024-01-15T14:36:55.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.299+0000 7f9168fd4700 1 -- 172.21.15.77:0/2661526399 wait complete. 2024-01-15T14:36:55.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.300+0000 7f9168fd4700 1 Processor -- start 2024-01-15T14:36:55.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.300+0000 7f9168fd4700 1 -- start start 2024-01-15T14:36:55.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 0x7f9164155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91640ce090 0x7f916415c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164155da0 con 0x7f9164073f90 2024-01-15T14:36:55.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164156f70 con 0x7f91640ce090 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916415f320 con 0x7f916415e2a0 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 0x7f9164155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 0x7f9164155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37084/0 (socket says 172.21.15.77:37084) 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 -- 172.21.15.77:0/3631240959 learned_addr learned my addr 172.21.15.77:0/3631240959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9161d9b700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91640ce090 0x7f916415c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9162d9d700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 -- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 msgr2=0x7f916415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916415b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 -- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91640ce090 msgr2=0x7f916415c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91640ce090 0x7f916415c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 -- 172.21.15.77:0/3631240959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154009550 con 0x7f9164073f90 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9161d9b700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91640ce090 0x7f916415c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9162d9d700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f916259c700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 0x7f9164155fa0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f915800e9b0 tx=0x7f915800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158009710 con 0x7f9164073f90 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915800c850 con 0x7f9164073f90 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91641569b0 con 0x7f9164073f90 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915800cbc0 con 0x7f9164073f90 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.301+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916415f570 con 0x7f9164073f90 2024-01-15T14:36:55.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.302+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9164067f60 con 0x7f9164073f90 2024-01-15T14:36:55.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.303+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f915800cd20 con 0x7f9164073f90 2024-01-15T14:36:55.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.303+0000 7f9152ffd700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f914406b590 0x7f914406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.303+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9158089dc0 con 0x7f9164073f90 2024-01-15T14:36:55.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.303+0000 7f9161d9b700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f914406b590 0x7f914406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.303+0000 7f9161d9b700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f914406b590 0x7f914406da50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f916414d0a0 tx=0x7f9154008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.308+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9158054bc0 con 0x7f9164073f90 2024-01-15T14:36:55.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.501+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91640cd240 con 0x7f9164073f90 2024-01-15T14:36:55.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.502+0000 7f9152ffd700 1 -- 172.21.15.77:0/3631240959 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13029 (secure 0 0 0) 0x7f91580581e0 con 0x7f9164073f90 2024-01-15T14:36:55.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-15T14:36:55.504 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"fsid":"a5eae108-559b-4366-a994-4d2898914163","created":"2024-01-15T14:34:39.303058+0000","modified":"2024-01-15T14:36:54.806269+0000","last_up_change":"2024-01-15T14:34:51.120790+0000","last_in_change":"2024-01-15T14:34:41.936699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T14:34:53.060193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d32a1a3d-1302-4ded-ba06-c91ff40bf18a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6825","nonce":2427925803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6827","nonce":2427925803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6831","nonce":2427925803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2427925803},{"type":"v1","addr":"172.21.15.77:6829","nonce":2427925803}]},"public_addr":"172.21.15.77:6825/2427925803","cluster_addr":"172.21.15.77:6827/2427925803","heartbeat_back_addr":"172.21.15.77:6831/2427925803","heartbeat_front_addr":"172.21.15.77:6829/2427925803","state":["exists","up"]},{"osd":1,"uuid":"4079025f-8ebd-4dc4-a2b2-146beab483f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6809","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6811","nonce":1232145209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6816","nonce":1232145209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6823","nonce":1232145209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1232145209},{"type":"v1","addr":"172.21.15.77:6820","nonce":1232145209}]},"public_addr":"172.21.15.77:6811/1232145209","cluster_addr":"172.21.15.77:6816/1232145209","heartbeat_back_addr":"172.21.15.77:6823/1232145209","heartbeat_front_addr":"172.21.15.77:6820/1232145209","state":["exists","up"]},{"osd":2,"uuid":"c22b946b-abe1-42da-b00e-2c272bef68a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6801","nonce":1734185656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6803","nonce":1734185656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6807","nonce":1734185656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1734185656},{"type":"v1","addr":"172.21.15.77:6805","nonce":1734185656}]},"public_addr":"172.21.15.77:6801/1734185656","cluster_addr":"172.21.15.77:6803/1734185656","heartbeat_back_addr":"172.21.15.77:6807/1734185656","heartbeat_front_addr":"172.21.15.77:6805/1734185656","state":["exists","up"]},{"osd":3,"uuid":"2874d645-bf49-4305-bdcb-8368258f238b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6810","nonce":3087735168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6814","nonce":3087735168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6821","nonce":3087735168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":3087735168},{"type":"v1","addr":"172.21.15.77:6817","nonce":3087735168}]},"public_addr":"172.21.15.77:6810/3087735168","cluster_addr":"172.21.15.77:6814/3087735168","heartbeat_back_addr":"172.21.15.77:6821/3087735168","heartbeat_front_addr":"172.21.15.77:6817/3087735168","state":["exists","up"]},{"osd":4,"uuid":"722287a8-22c1-4d04-92a0-2e538826f8f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6801","nonce":3726410932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6803","nonce":3726410932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6807","nonce":3726410932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3726410932},{"type":"v1","addr":"172.21.15.159:6805","nonce":3726410932}]},"public_addr":"172.21.15.159:6801/3726410932","cluster_addr":"172.21.15.159:6803/3726410932","heartbeat_back_addr":"172.21.15.159:6807/3726410932","heartbeat_front_addr":"172.21.15.159:6805/3726410932","state":["exists","up"]},{"osd":5,"uuid":"47ab64b1-a18f-42dd-875a-3df51f073d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6819","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6821","nonce":322902497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6823","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6825","nonce":322902497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6831","nonce":322902497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6827","nonce":322902497},{"type":"v1","addr":"172.21.15.159:6829","nonce":322902497}]},"public_addr":"172.21.15.159:6821/322902497","cluster_addr":"172.21.15.159:6825/322902497","heartbeat_back_addr":"172.21.15.159:6831/322902497","heartbeat_front_addr":"172.21.15.159:6829/322902497","state":["exists","up"]},{"osd":6,"uuid":"3c2335e5-12aa-4141-8e1d-628092b4dfb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6809","nonce":1865006471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6811","nonce":1865006471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6815","nonce":1865006471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1865006471},{"type":"v1","addr":"172.21.15.159:6813","nonce":1865006471}]},"public_addr":"172.21.15.159:6809/1865006471","cluster_addr":"172.21.15.159:6811/1865006471","heartbeat_back_addr":"172.21.15.159:6815/1865006471","heartbeat_front_addr":"172.21.15.159:6813/1865006471","state":["exists","up"]},{"osd":7,"uuid":"5154d400-3a7b-45d9-8e34-2d3057cabd8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6817","nonce":4215207465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6820","nonce":4215207465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6828","nonce":4215207465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":4215207465},{"type":"v1","addr":"172.21.15.159:6824","nonce":4215207465}]},"public_addr":"172.21.15.159:6817/4215207465","cluster_addr":"172.21.15.159:6820/4215207465","heartbeat_back_addr":"172.21.15.159:6828/4215207465","heartbeat_front_addr":"172.21.15.159:6824/4215207465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.157737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.143435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.092298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.131945+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.068947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.142200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.083989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T14:34:47.095609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/1676979021":"2024-01-16T14:36:51.787713+0000","172.21.15.77:6835/1676979021":"2024-01-16T14:36:51.787713+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T14:36:55.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f914406b590 msgr2=0x7f914406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f914406b590 0x7f914406da50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f916414d0a0 tx=0x7f9154008030 comp rx=0 tx=0).stop 2024-01-15T14:36:55.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 msgr2=0x7f9164155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 0x7f9164155fa0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f915800e9b0 tx=0x7f915800ed70 comp rx=0 tx=0).stop 2024-01-15T14:36:55.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 shutdown_connections 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f914406b590 0x7f914406da50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f916415e2a0 0x7f916415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9164073f90 0x7f9164155fa0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 --2- 172.21.15.77:0/3631240959 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f91640ce090 0x7f916415c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 >> 172.21.15.77:0/3631240959 conn(0x7f91640cd570 msgr2=0x7f91640cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 shutdown_connections 2024-01-15T14:36:55.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.504+0000 7f9168fd4700 1 -- 172.21.15.77:0/3631240959 wait complete. 2024-01-15T14:36:55.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T14:34:53.060193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T14:36:55.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T14:36:55.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.646+0000 7f0d28e64700 1 Processor -- start 2024-01-15T14:36:55.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.646+0000 7f0d28e64700 1 -- start start 2024-01-15T14:36:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d28e64700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f0d240b04f0 0x7f0d2415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d28e64700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240b0060 con 0x7f0d2406c710 2024-01-15T14:36:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d28e64700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2415f730 con 0x7f0d241445c0 2024-01-15T14:36:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d28e64700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d240af730 con 0x7f0d240b04f0 2024-01-15T14:36:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d22d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0d2406c710 0x7f0d2415ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49948/0 (socket says 172.21.15.77:49948) 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d22d9d700 1 -- 172.21.15.77:0/1635909544 learned_addr learned my addr 172.21.15.77:0/1635909544 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d21d9b700 1 --2- 172.21.15.77:0/1635909544 >> v2:172.21.15.159:3300/0 conn(0x7f0d240b04f0 0x7f0d2415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327734654 0 0) 0x7f0d240b0060 con 0x7f0d2406c710 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c0035b0 con 0x7f0d2406c710 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362461116 0 0) 0x7f0d2415f730 con 0x7f0d241445c0 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d240b0060 con 0x7f0d241445c0 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3747846310 0 0) 0x7f0d0c0035b0 con 0x7f0d2406c710 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 >> v2:172.21.15.159:3300/0 conn(0x7f0d240b04f0 msgr2=0x7f0d2415e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 --2- 172.21.15.77:0/1635909544 >> v2:172.21.15.159:3300/0 conn(0x7f0d240b04f0 0x7f0d2415e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 >> v1:172.21.15.159:6789/0 conn(0x7f0d241445c0 legacy=0x7f0d2406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240afa10 con 0x7f0d2406c710 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 361817355 0 0) 0x7f0d1c0019f0 con 0x7f0d2406c710 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 >> v1:172.21.15.77:6789/0 conn(0x7f0d2406c710 legacy=0x7f0d2415ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d0c003b90 0x7f0d0c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c004850 0x7f0d0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d0c005210 0x7f0d0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240af730 con 0x7f0d0c005210 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.647+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c00c090 con 0x7f0d0c003b90 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c00e460 con 0x7f0d0c004850 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d0c005210 0x7f0d0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d2259c700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d0c003b90 0x7f0d0c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d21d9b700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c004850 0x7f0d0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 -- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c004850 msgr2=0x7f0d0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c004850 0x7f0d0c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 -- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d0c003b90 msgr2=0x7f0d0c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d0c003b90 0x7f0d0c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 -- 172.21.15.77:0/1635909544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0035b0 con 0x7f0d0c005210 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 -- 172.21.15.77:0/1635909544 reap_dead start 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d22d9d700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d0c005210 0x7f0d0c00ba10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00a7a0 tx=0x7f0d0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b50b9e8b4795c9e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c012600 con 0x7f0d0c005210 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1c012780 con 0x7f0d0c005210 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.648+0000 7f0d2159a700 1 -- 172.21.15.77:0/1635909544 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c012b30 con 0x7f0d0c005210 2024-01-15T14:36:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 -- 172.21.15.77:0/1635909544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d0c005210 msgr2=0x7f0d0c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d0c005210 0x7f0d0c00ba10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00a7a0 tx=0x7f0d0c005880 comp rx=0 tx=0).stop 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 -- 172.21.15.77:0/1635909544 shutdown_connections 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 --2- 172.21.15.77:0/1635909544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d0c005210 0x7f0d0c00ba10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 -- 172.21.15.77:0/1635909544 >> 172.21.15.77:0/1635909544 conn(0x7f0d24071c10 msgr2=0x7f0d24071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 -- 172.21.15.77:0/1635909544 shutdown_connections 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.649+0000 7f0d28e64700 1 -- 172.21.15.77:0/1635909544 wait complete. 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.650+0000 7f0d28e64700 1 Processor -- start 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.650+0000 7f0d28e64700 1 -- start start 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d28e64700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c005210 0x7f0d2413e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d28e64700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 0x7f0d240b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d28e64700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d240cbd40 0x7f0d240cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d28e64700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240cc9c0 con 0x7f0d240b1970 2024-01-15T14:36:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d28e64700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2413f0d0 con 0x7f0d240cbd40 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d28e64700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240b2330 con 0x7f0d0c005210 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 0x7f0d240b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 0x7f0d240b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37112/0 (socket says 172.21.15.77:37112) 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 -- 172.21.15.77:0/3897209899 learned_addr learned my addr 172.21.15.77:0/3897209899 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d22d9d700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d240cbd40 0x7f0d240cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d2259c700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c005210 0x7f0d2413e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 -- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c005210 msgr2=0x7f0d2413e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c005210 0x7f0d2413e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 -- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d240cbd40 msgr2=0x7f0d240cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d240cbd40 0x7f0d240cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 -- 172.21.15.77:0/3897209899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2413e970 con 0x7f0d240b1970 2024-01-15T14:36:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.651+0000 7f0d21d9b700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 0x7f0d240b1df0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0d18011940 tx=0x7f0d18011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.652+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18004030 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.652+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18004190 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.652+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d241400a0 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.652+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18004560 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.652+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2413f2e0 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.653+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d24069020 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.653+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f0d1800f730 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.653+0000 7f0d12ffd700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0d0406b590 0x7f0d0406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.654+0000 7f0d2259c700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0d0406b590 0x7f0d0406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.654+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f0d1808a410 con 0x7f0d240b1970 2024-01-15T14:36:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.654+0000 7f0d2259c700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0d0406b590 0x7f0d0406da50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0d14000c00 tx=0x7f0d14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.658+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d18055200 con 0x7f0d240b1970 2024-01-15T14:36:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.846+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d2406f0b0 con 0x7f0d240b1970 2024-01-15T14:36:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.846+0000 7f0d12ffd700 1 -- 172.21.15.77:0/3897209899 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d2406f0b0 con 0x7f0d240b1970 2024-01-15T14:36:55.848 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-01-15T14:36:55.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0d0406b590 msgr2=0x7f0d0406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0d0406b590 0x7f0d0406da50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0d14000c00 tx=0x7f0d14009040 comp rx=0 tx=0).stop 2024-01-15T14:36:55.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 msgr2=0x7f0d240b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 0x7f0d240b1df0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0d18011940 tx=0x7f0d18011c50 comp rx=0 tx=0).stop 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 shutdown_connections 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f0d0406b590 0x7f0d0406da50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f0d0c005210 0x7f0d2413e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f0d240cbd40 0x7f0d240cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 --2- 172.21.15.77:0/3897209899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d240b1970 0x7f0d240b1df0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 >> 172.21.15.77:0/3897209899 conn(0x7f0d24071c10 msgr2=0x7f0d24141c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 shutdown_connections 2024-01-15T14:36:55.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.848+0000 7f0d28e64700 1 -- 172.21.15.77:0/3897209899 wait complete. 2024-01-15T14:36:55.862 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T14:36:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.985+0000 7f6b79ace700 1 Processor -- start 2024-01-15T14:36:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.985+0000 7f6b79ace700 1 -- start start 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b79ace700 1 --2- >> v2:172.21.15.159:3300/0 conn(0x7f6b7415e4b0 0x7f6b74073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b79ace700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7415eb00 con 0x7f6b741473a0 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b79ace700 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7415ede0 con 0x7f6b74147780 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b79ace700 1 -- --> v2:172.21.15.159:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7415e910 con 0x7f6b7415e4b0 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b737fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6b741473a0 0x7f6b7406c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49958/0 (socket says 172.21.15.77:49958) 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b737fe700 1 -- 172.21.15.77:0/3896250587 learned_addr learned my addr 172.21.15.77:0/3896250587 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b73fff700 1 --2- 172.21.15.77:0/3896250587 >> v2:172.21.15.159:3300/0 conn(0x7f6b7415e4b0 0x7f6b74073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234481466 0 0) 0x7f6b7415eb00 con 0x7f6b741473a0 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5c0035b0 con 0x7f6b741473a0 2024-01-15T14:36:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408262131 0 0) 0x7f6b7415ede0 con 0x7f6b74147780 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.986+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b7415eb00 con 0x7f6b74147780 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2357065363 0 0) 0x7f6b5c0035b0 con 0x7f6b741473a0 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b73fff700 1 -- 172.21.15.77:0/3896250587 >> v1:172.21.15.159:6789/0 conn(0x7f6b74147780 legacy=0x7f6b7406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b73fff700 1 -- 172.21.15.77:0/3896250587 >> v1:172.21.15.77:6789/0 conn(0x7f6b741473a0 legacy=0x7f6b7406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b73fff700 1 -- 172.21.15.77:0/3896250587 --> v2:172.21.15.159:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7415f0c0 con 0x7f6b7415e4b0 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b73fff700 1 --2- 172.21.15.77:0/3896250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b7415e4b0 0x7f6b74073f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d90 tx=0x7f6b6c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44ced2d9b9af8c8c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 <== mon.1 v2:172.21.15.159:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00e070 con 0x7f6b7415e4b0 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 <== mon.1 v2:172.21.15.159:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c003a20 con 0x7f6b7415e4b0 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b727fc700 1 -- 172.21.15.77:0/3896250587 <== mon.1 v2:172.21.15.159:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68001670 con 0x7f6b7415e4b0 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b79ace700 1 -- 172.21.15.77:0/3896250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b7415e4b0 msgr2=0x7f6b74073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3896250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b7415e4b0 0x7f6b74073f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d90 tx=0x7f6b6c00b330 comp rx=0 tx=0).stop 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b79ace700 1 -- 172.21.15.77:0/3896250587 shutdown_connections 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.987+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3896250587 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b7415e4b0 0x7f6b74073f90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.988+0000 7f6b79ace700 1 -- 172.21.15.77:0/3896250587 >> 172.21.15.77:0/3896250587 conn(0x7f6b740b3130 msgr2=0x7f6b740d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.988+0000 7f6b79ace700 1 -- 172.21.15.77:0/3896250587 shutdown_connections 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.988+0000 7f6b79ace700 1 -- 172.21.15.77:0/3896250587 wait complete. 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.989+0000 7f6b79ace700 1 Processor -- start 2024-01-15T14:36:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.989+0000 7f6b79ace700 1 -- start start 2024-01-15T14:36:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 0x7f6b741f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 --2- >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b74147780 0x7f6b740ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 --2- >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b7415e4b0 0x7f6b740ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740ae280 con 0x7f6b741473a0 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 -- --> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b74072530 con 0x7f6b74147780 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 -- --> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740ae490 con 0x7f6b7415e4b0 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 0x7f6b741f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 0x7f6b741f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37126/0 (socket says 172.21.15.77:37126) 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 -- 172.21.15.77:0/3932179562 learned_addr learned my addr 172.21.15.77:0/3932179562 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-15T14:36:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 -- 172.21.15.77:0/3932179562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b7415e4b0 msgr2=0x7f6b740ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b7415e4b0 0x7f6b740ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 -- 172.21.15.77:0/3932179562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b74147780 msgr2=0x7f6b740ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b74147780 0x7f6b740ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 -- 172.21.15.77:0/3932179562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c009a70 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b737fe700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 0x7f6b741f30c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c770 tx=0x7f6b6800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68007560 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b680076c0 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b740ae770 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68004a70 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.990+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b740aed30 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.991+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b74067f60 con 0x7f6b741473a0 2024-01-15T14:36:55.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.992+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f6b68004bd0 con 0x7f6b741473a0 2024-01-15T14:36:55.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.992+0000 7f6b5bfff700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6b6006b590 0x7f6b6006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T14:36:55.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.992+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6b680892b0 con 0x7f6b741473a0 2024-01-15T14:36:55.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.992+0000 7f6b72ffd700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6b6006b590 0x7f6b6006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T14:36:55.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.993+0000 7f6b72ffd700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6b6006b590 0x7f6b6006da50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6b640010e0 tx=0x7f6b640079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T14:36:55.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:55.997+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b68054130 con 0x7f6b741473a0 2024-01-15T14:36:56.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.204+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6b7406c240 con 0x7f6b741473a0 2024-01-15T14:36:56.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.204+0000 7f6b5bfff700 1 -- 172.21.15.77:0/3932179562 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v37) v1 ==== 80+0+122 (secure 0 0 0) 0x7f6b68018090 con 0x7f6b741473a0 2024-01-15T14:36:56.205 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6837/1676979021 2024-01-16T14:36:51.787713+0000 2024-01-15T14:36:56.206 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6835/1676979021 2024-01-16T14:36:51.787713+0000 2024-01-15T14:36:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6b6006b590 msgr2=0x7f6b6006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6b6006b590 0x7f6b6006da50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6b640010e0 tx=0x7f6b640079e0 comp rx=0 tx=0).stop 2024-01-15T14:36:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 msgr2=0x7f6b741f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T14:36:56.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 0x7f6b741f30c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c770 tx=0x7f6b6800cb30 comp rx=0 tx=0).stop 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 shutdown_connections 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:6832/4256185546,v1:172.21.15.77:6833/4256185546] conn(0x7f6b6006b590 0x7f6b6006da50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.159:3301/0,v1:172.21.15.159:6790/0] conn(0x7f6b7415e4b0 0x7f6b740ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b741473a0 0x7f6b741f30c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 --2- 172.21.15.77:0/3932179562 >> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] conn(0x7f6b74147780 0x7f6b740ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 >> 172.21.15.77:0/3932179562 conn(0x7f6b740b3130 msgr2=0x7f6b741466c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 shutdown_connections 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-15T14:36:56.206+0000 7f6b79ace700 1 -- 172.21.15.77:0/3932179562 wait complete. 2024-01-15T14:36:56.209 INFO:teuthology.orchestra.run.smithi077.stderr:listed 2 entries 2024-01-15T14:36:56.220 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-15T14:36:56.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T14:36:56.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-15T14:37:03.358 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:37:03.361 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:37:33.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:37:33.431 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:38:03.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:38:03.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:38:33.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:38:33.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:39:03.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:39:03.667 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:39:33.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:39:33.755 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:40:03.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:40:03.827 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:40:33.895 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:40:33.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:41:03.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:41:03.970 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:41:34.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:41:34.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:41:56.225 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-01-15T14:41:56.227 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-01-15T14:42:04.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:42:04.164 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:42:34.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:42:34.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:43:04.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:43:04.300 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:43:34.369 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:43:34.372 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:44:04.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:44:04.461 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:44:34.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:44:34.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:45:04.625 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:45:04.629 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:45:34.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:45:34.717 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:46:04.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:46:04.790 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:46:34.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:46:34.862 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:47:04.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:47:04.930 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:47:35.001 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:47:35.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:48:05.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:48:05.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:48:35.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:48:35.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:49:05.234 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:49:05.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:49:35.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:49:35.327 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:50:05.421 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:50:05.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:50:35.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:50:35.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:51:05.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:51:05.588 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:51:35.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:51:35.665 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:52:05.803 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:52:05.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:52:35.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:52:35.882 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:53:05.948 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:53:05.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:53:36.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:53:36.069 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:54:06.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:54:06.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:54:36.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:54:36.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:55:06.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:55:06.308 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:55:36.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:55:36.381 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:56:06.466 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:56:06.471 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:56:36.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:56:36.551 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:57:06.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:57:06.619 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:57:36.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:57:36.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:58:06.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:58:06.781 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:58:36.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:58:36.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:59:06.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:59:06.939 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:59:37.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T14:59:37.095 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:00:07.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:00:07.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:00:37.317 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:00:37.321 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:01:07.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:01:07.401 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:01:37.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:01:37.543 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:02:07.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:02:07.671 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:02:37.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:02:37.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:03:07.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:03:07.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:03:38.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:03:38.061 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:04:08.212 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:04:08.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:04:38.533 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:04:38.536 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:05:08.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:05:08.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:05:38.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:05:38.722 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:06:08.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:06:08.821 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:06:38.953 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:06:38.956 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:07:09.117 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:07:09.124 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:07:39.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:07:39.201 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:08:09.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:08:09.324 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:08:39.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:08:39.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:09:09.462 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:09:09.466 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:09:39.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:09:39.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:10:09.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:10:09.676 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:10:39.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:10:39.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:11:09.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:11:09.987 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:11:40.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:11:40.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:12:10.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:12:10.268 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:12:40.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:12:40.394 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:13:10.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:13:10.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:13:40.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:13:40.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:14:10.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:14:10.680 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:14:40.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:14:40.824 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:15:10.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:15:10.988 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:15:41.053 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:15:41.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:16:11.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:16:11.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:16:41.251 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:16:41.254 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:17:11.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:17:11.402 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:17:41.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:17:41.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:18:11.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:18:11.637 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:18:41.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:18:41.708 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:19:11.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:19:11.820 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:19:41.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:19:41.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:20:12.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:20:12.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:20:42.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:20:42.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:21:12.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:21:12.423 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:21:42.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:21:42.545 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:22:12.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:22:12.684 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:22:42.752 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:22:42.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:23:12.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:23:12.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:23:43.045 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:23:43.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:24:13.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:24:13.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:24:43.314 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:24:43.318 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:25:13.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:25:13.492 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:25:43.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:25:43.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:26:13.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:26:13.673 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:26:43.833 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:26:43.836 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:27:13.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:27:13.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:27:44.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:27:44.054 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:28:14.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:28:14.209 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:28:44.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:28:44.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:29:14.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:29:14.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:29:44.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:29:44.699 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:30:14.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:30:14.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:30:44.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:30:44.967 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:31:15.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:31:15.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:31:45.242 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:31:45.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:32:15.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:32:15.381 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:32:45.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:32:45.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:33:15.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:33:15.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:33:45.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:33:45.824 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:34:15.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:34:15.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:34:46.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:34:46.105 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:35:16.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:35:16.191 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:35:46.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:35:46.356 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:36:16.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:36:16.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:36:46.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:36:46.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:37:16.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:37:16.789 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:37:46.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:37:46.961 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:38:17.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:38:17.113 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:38:47.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:38:47.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:39:17.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:39:17.251 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:39:47.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:39:47.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:40:17.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:40:17.619 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:40:47.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:40:47.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:41:17.904 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:41:17.909 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:41:48.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:41:48.051 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:42:18.195 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:42:18.198 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:42:48.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:42:48.360 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:43:18.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:43:18.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:43:48.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:43:48.668 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:44:18.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:44:18.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:44:48.994 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:44:48.998 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:45:19.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:45:19.145 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:45:49.240 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:45:49.244 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:46:19.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:46:19.397 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:46:49.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:46:49.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:47:19.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:47:19.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:47:49.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:47:49.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:48:19.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:48:19.909 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:48:50.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:48:50.128 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:49:20.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:49:20.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:49:50.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:49:50.339 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:50:20.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:50:20.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:50:50.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:50:50.651 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:51:20.761 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:51:20.765 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:51:50.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:51:50.931 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:52:21.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:52:21.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:52:51.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:52:51.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:53:21.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:53:21.443 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:53:51.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:53:51.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:54:21.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:54:21.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:54:52.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:54:52.069 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:55:22.235 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:55:22.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:55:52.365 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:55:52.369 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:56:22.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:56:22.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:56:52.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:56:52.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:57:22.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:57:22.844 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:57:52.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:57:52.975 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:58:23.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:58:23.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:58:53.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:58:53.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:59:23.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:59:23.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:59:53.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T15:59:53.555 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:00:23.723 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:00:23.726 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:00:53.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:00:53.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:01:23.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:01:23.991 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:01:54.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:01:54.126 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:02:24.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:02:24.285 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:02:54.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:02:54.440 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:03:24.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:03:24.586 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:03:54.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:03:54.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:04:24.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:04:24.832 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:04:54.977 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:04:54.982 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:05:25.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:05:25.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:05:55.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:05:55.283 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:06:25.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:06:25.387 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:06:55.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:06:55.560 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:07:25.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:07:25.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:07:55.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:07:55.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:08:25.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:08:25.993 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:08:56.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:08:56.256 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:09:26.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:09:26.395 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:09:56.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:09:56.501 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:10:26.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:10:26.657 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:10:56.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:10:56.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:11:27.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:11:27.018 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:11:57.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:11:57.145 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:12:27.242 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:12:27.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:12:57.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:12:57.376 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:13:27.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:13:27.524 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:13:57.650 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:13:57.655 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:14:27.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:14:27.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:14:57.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:14:57.914 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:15:28.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:15:28.087 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:15:58.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:15:58.226 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:16:28.362 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:16:28.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:16:58.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:16:58.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:17:28.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:17:28.601 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:17:58.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:17:58.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:18:28.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:18:28.829 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:18:58.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:18:58.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:19:29.038 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:19:29.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:19:59.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:19:59.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:20:29.283 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:20:29.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:20:59.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:20:59.442 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:21:29.570 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:21:29.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:21:59.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:21:59.640 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:22:29.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:22:29.711 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:22:59.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:22:59.850 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:23:29.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:23:29.996 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:24:00.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:24:00.169 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:24:30.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:24:30.316 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:25:00.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:25:00.555 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:25:30.632 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:25:30.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:26:00.761 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:26:00.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:26:30.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:26:30.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:27:01.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:27:01.060 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:27:31.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:27:31.223 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:28:01.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:28:01.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:28:31.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:28:31.473 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:29:01.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:29:01.600 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:29:31.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:29:31.730 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:30:01.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:30:01.852 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:30:32.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:30:32.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:31:02.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:31:02.131 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:31:32.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:31:32.270 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:32:02.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:32:02.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:32:32.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:32:32.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:33:02.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:33:02.712 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:33:32.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:33:32.809 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:34:03.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:34:03.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:34:33.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:34:33.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:35:03.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:35:03.442 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:35:33.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:35:33.598 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:36:03.667 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:36:03.671 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:36:33.804 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:36:33.808 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:37:03.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:37:03.965 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:37:34.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:37:34.113 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:38:04.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:38:04.270 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:38:34.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:38:34.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:39:04.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:39:04.564 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:39:34.675 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:39:34.678 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:40:04.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:40:04.754 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:40:34.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:40:34.824 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:41:04.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:41:04.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:41:35.082 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:41:35.086 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:42:05.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:42:05.225 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:42:35.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:42:35.376 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:43:05.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:43:05.524 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:43:35.665 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:43:35.668 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:44:05.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:44:05.815 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:44:35.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:44:35.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:45:06.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:45:06.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:45:36.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:45:36.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:46:06.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:46:06.367 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:46:36.503 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:46:36.507 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:47:06.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:47:06.662 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:47:36.864 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:47:36.868 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:48:06.953 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:48:06.957 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:48:37.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:48:37.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:49:07.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:49:07.278 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:49:37.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:49:37.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:50:07.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:50:07.548 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:50:37.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:50:37.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:51:07.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:51:07.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:51:37.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:51:37.988 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:52:08.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:52:08.111 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:52:38.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:52:38.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:53:08.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:53:08.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:53:38.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:53:38.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:54:08.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:54:08.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:54:38.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:54:38.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:55:09.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:55:09.018 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:55:39.170 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:55:39.174 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:56:09.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:56:09.295 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:56:39.421 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:56:39.424 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:57:09.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:57:09.546 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:57:39.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:57:39.676 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:58:09.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:58:09.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:58:39.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:58:39.996 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:59:10.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:59:10.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:59:40.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T16:59:40.219 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:00:10.347 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:00:10.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:00:40.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:00:40.515 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:01:10.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:01:10.640 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:01:40.799 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:01:40.803 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:02:10.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:02:10.950 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:02:41.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:02:41.101 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:03:11.241 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:03:11.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:03:41.388 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:03:41.391 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:04:11.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:04:11.511 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:04:41.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:04:41.666 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:05:11.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:05:11.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:05:41.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:05:41.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:06:12.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:06:12.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:06:42.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:06:42.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:07:12.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:07:12.327 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:07:42.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:07:42.475 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:08:12.618 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:08:12.622 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:08:42.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:08:42.785 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:09:12.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:09:12.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:09:43.053 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:09:43.058 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:10:13.155 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:10:13.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:10:43.346 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:10:43.350 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:11:13.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:11:13.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:11:43.485 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:11:43.489 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:12:13.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:12:13.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:12:43.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:12:43.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:13:13.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:13:13.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:13:43.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:13:43.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:14:14.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:14:14.100 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:14:44.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:14:44.206 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:15:14.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:15:14.321 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:15:44.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:15:44.486 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:16:14.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:16:14.625 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:16:44.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:16:44.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:17:15.011 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:17:15.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:17:45.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:17:45.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:18:15.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:18:15.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:18:45.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:18:45.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:19:15.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:19:15.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:19:45.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:19:45.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:20:15.835 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:20:15.838 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:20:45.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:20:45.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:21:16.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:21:16.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:21:46.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:21:46.186 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:22:16.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:22:16.334 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:22:46.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:22:46.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:23:16.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:23:16.662 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:23:46.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:23:46.809 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:24:16.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:24:16.938 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:24:47.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:24:47.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:25:17.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:25:17.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:25:47.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:25:47.278 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:26:17.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:26:17.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:26:47.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:26:47.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:27:17.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:27:17.672 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:27:47.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:27:47.810 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:28:17.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:28:17.947 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:28:48.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:28:48.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:29:18.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:29:18.263 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:29:48.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:29:48.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:30:18.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:30:18.535 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:30:48.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:30:48.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:31:18.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:31:18.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:31:49.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:31:49.050 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:32:12.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-01-15T17:32:12.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-01-15T17:32:12.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-15T17:32:12.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-15T17:32:12.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-15T17:32:12.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:32:12.446 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.0... 2024-01-15T17:32:12.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:32:12.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:32:12.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:32:17.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-15T17:32:17.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-15T17:32:17.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-15T17:32:17.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-15T17:32:17.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-15T17:32:17.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~6s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~6s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-15T17:32:17.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~6s 2024-01-15T17:32:19.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:32:19.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:32:21.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:32:21.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:32:22.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-01-15T17:32:22.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-01-15T17:32:22.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-01-15T17:32:22.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~11s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~11s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~11s 2024-01-15T17:32:22.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~11s 2024-01-15T17:32:22.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~5s 2024-01-15T17:32:22.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~11s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~10s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~10s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~10s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~10s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~10s 2024-01-15T17:32:27.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~10s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~10s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~10s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~16s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~16s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~10s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~10s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~16s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~16s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~10s 2024-01-15T17:32:27.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~16s 2024-01-15T17:32:32.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~15s 2024-01-15T17:32:32.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~15s 2024-01-15T17:32:32.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~21s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~21s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~21s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~21s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~15s 2024-01-15T17:32:32.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~21s 2024-01-15T17:32:37.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~20s 2024-01-15T17:32:37.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~26s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~26s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~20s 2024-01-15T17:32:37.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~26s 2024-01-15T17:32:37.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~26s 2024-01-15T17:32:37.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~20s 2024-01-15T17:32:37.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~26s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~25s 2024-01-15T17:32:42.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~25s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~31s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~31s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~25s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~25s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~31s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~31s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~25s 2024-01-15T17:32:42.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~31s 2024-01-15T17:32:47.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~30s 2024-01-15T17:32:47.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~30s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~36s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~36s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~30s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~30s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~36s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~36s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~30s 2024-01-15T17:32:47.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~36s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~35s 2024-01-15T17:32:52.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~35s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~41s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~41s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~35s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~35s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~41s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~41s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~35s 2024-01-15T17:32:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~41s 2024-01-15T17:32:57.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~40s 2024-01-15T17:32:57.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~40s 2024-01-15T17:32:57.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~46s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~46s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~40s 2024-01-15T17:32:57.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~46s 2024-01-15T17:32:57.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~46s 2024-01-15T17:32:57.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~40s 2024-01-15T17:32:57.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~46s 2024-01-15T17:33:02.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~51s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~51s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~51s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~51s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~45s 2024-01-15T17:33:02.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~51s 2024-01-15T17:33:07.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~50s 2024-01-15T17:33:07.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~56s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~56s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~50s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~56s 2024-01-15T17:33:07.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~56s 2024-01-15T17:33:07.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~50s 2024-01-15T17:33:07.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~56s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~55s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~55s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~55s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~55s 2024-01-15T17:33:12.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~55s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~55s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~55s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~61s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~61s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~55s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~55s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~61s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~61s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~55s 2024-01-15T17:33:12.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~61s 2024-01-15T17:33:17.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~60s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~66s 2024-01-15T17:33:17.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~66s 2024-01-15T17:33:17.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2024-01-15T17:33:17.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~60s 2024-01-15T17:33:17.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~66s 2024-01-15T17:33:17.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~66s 2024-01-15T17:33:17.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~60s 2024-01-15T17:33:17.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~66s 2024-01-15T17:33:19.204 DEBUG:teuthology.orchestra.remote:timed out 2024-01-15T17:33:21.459 DEBUG:teuthology.orchestra.remote:timed out 2024-01-15T17:33:22.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~65s 2024-01-15T17:33:22.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-01-15T17:33:22.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~71s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~71s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~65s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~71s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~71s 2024-01-15T17:33:22.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~65s 2024-01-15T17:33:22.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~71s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~70s 2024-01-15T17:33:27.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~70s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~76s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~76s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~70s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~70s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~76s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~76s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~70s 2024-01-15T17:33:27.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~76s 2024-01-15T17:33:28.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:28.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:31.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:32.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~75s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~81s 2024-01-15T17:33:32.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~81s 2024-01-15T17:33:32.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~75s 2024-01-15T17:33:32.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~75s 2024-01-15T17:33:32.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~81s 2024-01-15T17:33:32.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~81s 2024-01-15T17:33:32.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~75s 2024-01-15T17:33:32.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~81s 2024-01-15T17:33:33.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:33.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:34.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:37.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:37.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:37.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~80s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~86s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~86s 2024-01-15T17:33:37.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~80s 2024-01-15T17:33:37.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~80s 2024-01-15T17:33:37.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~86s 2024-01-15T17:33:37.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~86s 2024-01-15T17:33:37.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~80s 2024-01-15T17:33:37.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~86s 2024-01-15T17:33:40.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:40.413 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:33:40.418 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ERROR 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner:ERROR: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T17:33:40.419 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 160, in umount 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T17:33:40.420 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 148, in wait 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: for line in f: 2024-01-15T17:33:40.421 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner:socket.timeout 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T17:33:40.422 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7609, in tearDown 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: super(self.__class__, self).tearDown() 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 459, in tearDown 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/cephfs_test_case.py", line 205, in tearDown 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: m.teardown() 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 207, in teardown 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: self.umount() 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 163, in umount 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: self.client_remote.run( 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner:ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:33:40.423 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:Ran 1 test in 10686.826s 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:ERROR: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T17:33:40.424 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 160, in umount 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T17:33:40.425 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 148, in wait 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: for line in f: 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-01-15T17:33:40.426 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner:socket.timeout 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7609, in tearDown 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: super(self.__class__, self).tearDown() 2024-01-15T17:33:40.427 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 459, in tearDown 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/cephfs_test_case.py", line 205, in tearDown 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: m.teardown() 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 207, in teardown 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: self.umount() 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 163, in umount 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: self.client_remote.run( 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner:ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:33:40.428 INFO:tasks.cephfs_test_runner: 2024-01-15T17:33:40.429 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T17:33:40.653 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9d008ac1c66c426986dcc1c80b6608cf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T17:33:40.654 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-15T17:33:40.662 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-01-15T17:33:40.674 INFO:tasks.kclient:Unmounting kernel clients... 2024-01-15T17:33:40.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:40.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:42.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-01-15T17:33:42.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2024-01-15T17:33:42.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~85s 2024-01-15T17:33:42.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~85s 2024-01-15T17:33:42.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~85s 2024-01-15T17:33:42.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~85s 2024-01-15T17:33:42.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~85s 2024-01-15T17:33:42.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~85s 2024-01-15T17:33:42.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~85s 2024-01-15T17:33:42.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~91s 2024-01-15T17:33:42.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~91s 2024-01-15T17:33:42.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~85s 2024-01-15T17:33:42.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~85s 2024-01-15T17:33:42.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~91s 2024-01-15T17:33:42.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~91s 2024-01-15T17:33:42.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~85s 2024-01-15T17:33:42.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~91s 2024-01-15T17:33:43.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:43.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:43.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:43.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:46.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:46.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:46.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:46.556 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:33:46.557 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 184, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:33:47.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2024-01-15T17:33:47.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2024-01-15T17:33:47.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2024-01-15T17:33:47.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~90s 2024-01-15T17:33:47.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~90s 2024-01-15T17:33:47.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~90s 2024-01-15T17:33:47.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~90s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~90s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~90s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~96s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~96s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~90s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~90s 2024-01-15T17:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~96s 2024-01-15T17:33:47.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~96s 2024-01-15T17:33:47.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~90s 2024-01-15T17:33:47.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~96s 2024-01-15T17:33:52.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-01-15T17:33:52.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2024-01-15T17:33:52.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~95s 2024-01-15T17:33:52.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~95s 2024-01-15T17:33:52.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~95s 2024-01-15T17:33:52.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2024-01-15T17:33:52.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~95s 2024-01-15T17:33:52.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~95s 2024-01-15T17:33:52.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~95s 2024-01-15T17:33:52.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~101s 2024-01-15T17:33:52.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~101s 2024-01-15T17:33:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~95s 2024-01-15T17:33:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~95s 2024-01-15T17:33:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~101s 2024-01-15T17:33:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~101s 2024-01-15T17:33:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~95s 2024-01-15T17:33:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~101s 2024-01-15T17:33:52.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:33:52.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:33:55.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:33:57.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~100s 2024-01-15T17:33:57.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~100s 2024-01-15T17:33:57.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~100s 2024-01-15T17:33:57.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~100s 2024-01-15T17:33:57.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~100s 2024-01-15T17:33:57.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~100s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~100s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~100s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~100s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~106s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~106s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~100s 2024-01-15T17:33:57.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~100s 2024-01-15T17:33:57.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~106s 2024-01-15T17:33:57.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~106s 2024-01-15T17:33:57.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~100s 2024-01-15T17:33:57.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~106s 2024-01-15T17:34:02.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~105s 2024-01-15T17:34:02.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2024-01-15T17:34:02.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2024-01-15T17:34:02.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~105s 2024-01-15T17:34:02.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~105s 2024-01-15T17:34:02.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~105s 2024-01-15T17:34:02.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~105s 2024-01-15T17:34:02.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~105s 2024-01-15T17:34:02.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~105s 2024-01-15T17:34:02.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~111s 2024-01-15T17:34:02.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~111s 2024-01-15T17:34:02.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~105s 2024-01-15T17:34:02.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~105s 2024-01-15T17:34:02.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~111s 2024-01-15T17:34:02.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~111s 2024-01-15T17:34:02.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~105s 2024-01-15T17:34:02.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~111s 2024-01-15T17:34:07.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~110s 2024-01-15T17:34:07.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~110s 2024-01-15T17:34:07.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~116s 2024-01-15T17:34:07.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~116s 2024-01-15T17:34:07.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~110s 2024-01-15T17:34:07.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~110s 2024-01-15T17:34:07.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~116s 2024-01-15T17:34:07.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~116s 2024-01-15T17:34:07.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~110s 2024-01-15T17:34:07.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~116s 2024-01-15T17:34:07.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:07.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:10.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:12.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~115s 2024-01-15T17:34:12.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~115s 2024-01-15T17:34:12.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~121s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~121s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~115s 2024-01-15T17:34:12.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~115s 2024-01-15T17:34:12.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~121s 2024-01-15T17:34:12.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~121s 2024-01-15T17:34:12.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~115s 2024-01-15T17:34:12.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~121s 2024-01-15T17:34:13.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:13.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:13.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:13.692 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:34:13.693 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:34:13.699 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-15T17:34:13.710 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T17:34:13.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:13.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:16.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:17.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~120s 2024-01-15T17:34:17.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2024-01-15T17:34:17.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~120s 2024-01-15T17:34:17.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2024-01-15T17:34:17.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~120s 2024-01-15T17:34:17.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~120s 2024-01-15T17:34:17.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~120s 2024-01-15T17:34:17.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~120s 2024-01-15T17:34:17.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~120s 2024-01-15T17:34:17.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~126s 2024-01-15T17:34:17.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~126s 2024-01-15T17:34:17.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~120s 2024-01-15T17:34:17.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~120s 2024-01-15T17:34:17.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~126s 2024-01-15T17:34:17.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~126s 2024-01-15T17:34:17.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~120s 2024-01-15T17:34:17.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~126s 2024-01-15T17:34:22.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~125s 2024-01-15T17:34:22.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2024-01-15T17:34:22.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~125s 2024-01-15T17:34:22.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~125s 2024-01-15T17:34:22.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~125s 2024-01-15T17:34:22.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~125s 2024-01-15T17:34:22.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~125s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~125s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~125s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~131s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~131s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~125s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~125s 2024-01-15T17:34:22.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~131s 2024-01-15T17:34:22.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~131s 2024-01-15T17:34:22.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~125s 2024-01-15T17:34:22.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~131s 2024-01-15T17:34:25.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:25.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:27.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2024-01-15T17:34:27.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-01-15T17:34:27.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2024-01-15T17:34:27.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2024-01-15T17:34:27.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2024-01-15T17:34:27.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~130s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~130s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~130s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~136s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~136s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~130s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~130s 2024-01-15T17:34:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~136s 2024-01-15T17:34:27.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~136s 2024-01-15T17:34:27.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~130s 2024-01-15T17:34:27.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~136s 2024-01-15T17:34:28.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:32.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2024-01-15T17:34:32.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-01-15T17:34:32.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2024-01-15T17:34:32.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~135s 2024-01-15T17:34:32.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~135s 2024-01-15T17:34:32.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~135s 2024-01-15T17:34:32.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~135s 2024-01-15T17:34:32.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~135s 2024-01-15T17:34:32.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~135s 2024-01-15T17:34:32.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~141s 2024-01-15T17:34:32.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~141s 2024-01-15T17:34:32.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~135s 2024-01-15T17:34:32.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~135s 2024-01-15T17:34:32.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~141s 2024-01-15T17:34:32.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~141s 2024-01-15T17:34:32.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~135s 2024-01-15T17:34:32.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~141s 2024-01-15T17:34:37.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~140s 2024-01-15T17:34:37.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-01-15T17:34:37.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2024-01-15T17:34:37.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~140s 2024-01-15T17:34:37.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~140s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~140s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~140s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~140s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~140s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~146s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~146s 2024-01-15T17:34:37.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~140s 2024-01-15T17:34:37.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~140s 2024-01-15T17:34:37.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~146s 2024-01-15T17:34:37.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~146s 2024-01-15T17:34:37.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~140s 2024-01-15T17:34:37.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~146s 2024-01-15T17:34:40.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:40.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:42.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~145s 2024-01-15T17:34:42.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~145s 2024-01-15T17:34:42.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~145s 2024-01-15T17:34:42.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~145s 2024-01-15T17:34:42.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~145s 2024-01-15T17:34:42.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~145s 2024-01-15T17:34:42.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~145s 2024-01-15T17:34:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~145s 2024-01-15T17:34:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~145s 2024-01-15T17:34:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~151s 2024-01-15T17:34:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~151s 2024-01-15T17:34:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~145s 2024-01-15T17:34:42.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~145s 2024-01-15T17:34:42.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~151s 2024-01-15T17:34:42.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~151s 2024-01-15T17:34:42.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~145s 2024-01-15T17:34:42.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~151s 2024-01-15T17:34:43.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:46.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:46.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:46.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:46.972 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:34:46.972 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:34:46.986 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-15T17:34:46.986 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T17:34:46.986 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:46.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:46.989 ERROR:teuthology.misc:Saw exception from ceph.mds.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-01-15T17:34:47.005 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T17:34:47.005 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.006 ERROR:teuthology.misc:Saw exception from ceph.mds.c Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-01-15T17:34:47.019 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T17:34:47.019 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.020 ERROR:teuthology.misc:Saw exception from ceph.mds.b Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2024-01-15T17:34:47.033 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T17:34:47.033 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.033 ERROR:teuthology.misc:Saw exception from ceph.mds.d Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2024-01-15T17:34:47.045 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-15T17:34:47.046 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-15T17:34:47.046 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.046 ERROR:teuthology.misc:Saw exception from ceph.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' 2024-01-15T17:34:47.065 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-15T17:34:47.065 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.065 ERROR:teuthology.misc:Saw exception from ceph.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' 2024-01-15T17:34:47.081 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-15T17:34:47.081 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.082 ERROR:teuthology.misc:Saw exception from ceph.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2' 2024-01-15T17:34:47.100 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-15T17:34:47.100 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.100 ERROR:teuthology.misc:Saw exception from ceph.osd.3 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3' 2024-01-15T17:34:47.117 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-15T17:34:47.117 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.117 ERROR:teuthology.misc:Saw exception from ceph.osd.4 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4' 2024-01-15T17:34:47.134 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-15T17:34:47.134 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.135 ERROR:teuthology.misc:Saw exception from ceph.osd.5 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5' 2024-01-15T17:34:47.153 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-15T17:34:47.153 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.154 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.154 ERROR:teuthology.misc:Saw exception from ceph.osd.6 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6' 2024-01-15T17:34:47.172 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-15T17:34:47.172 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.172 ERROR:teuthology.misc:Saw exception from ceph.osd.7 Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2024-01-15T17:34:47.192 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-15T17:34:47.192 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-15T17:34:47.192 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.192 ERROR:teuthology.misc:Saw exception from ceph.mgr.y Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-01-15T17:34:47.217 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-15T17:34:47.218 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.218 ERROR:teuthology.misc:Saw exception from ceph.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2024-01-15T17:34:47.242 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-15T17:34:47.242 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-15T17:34:47.242 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.242 ERROR:teuthology.misc:Saw exception from ceph.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi077 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a' 2024-01-15T17:34:47.271 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-15T17:34:47.271 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.272 ERROR:teuthology.misc:Saw exception from ceph.mon.b Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b' 2024-01-15T17:34:47.301 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-15T17:34:47.301 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:34:47.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T17:34:47.301 ERROR:teuthology.misc:Saw exception from ceph.mon.c Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi159 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c' 2024-01-15T17:34:47.328 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-15T17:34:47.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:47.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~150s 2024-01-15T17:34:47.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~150s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~150s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~156s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~156s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~150s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~150s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~156s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~156s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~150s 2024-01-15T17:34:47.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~156s 2024-01-15T17:34:50.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:34:52.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~155s 2024-01-15T17:34:52.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~155s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~161s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~161s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~155s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~155s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~161s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~161s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~155s 2024-01-15T17:34:52.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~161s 2024-01-15T17:34:57.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2024-01-15T17:34:57.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~160s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~166s 2024-01-15T17:34:57.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~166s 2024-01-15T17:34:57.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~160s 2024-01-15T17:34:57.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~160s 2024-01-15T17:34:57.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~166s 2024-01-15T17:34:57.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~166s 2024-01-15T17:34:57.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~160s 2024-01-15T17:34:57.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~166s 2024-01-15T17:34:59.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:34:59.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:02.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:02.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2024-01-15T17:35:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2024-01-15T17:35:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2024-01-15T17:35:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2024-01-15T17:35:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~165s 2024-01-15T17:35:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~165s 2024-01-15T17:35:02.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~165s 2024-01-15T17:35:02.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~165s 2024-01-15T17:35:02.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~165s 2024-01-15T17:35:02.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~171s 2024-01-15T17:35:02.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~171s 2024-01-15T17:35:02.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~165s 2024-01-15T17:35:02.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~165s 2024-01-15T17:35:02.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~171s 2024-01-15T17:35:02.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~171s 2024-01-15T17:35:02.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~165s 2024-01-15T17:35:02.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~171s 2024-01-15T17:35:07.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-01-15T17:35:07.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~170s 2024-01-15T17:35:07.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~170s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~176s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~176s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~170s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~170s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~176s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~176s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~170s 2024-01-15T17:35:07.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~176s 2024-01-15T17:35:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2024-01-15T17:35:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2024-01-15T17:35:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~175s 2024-01-15T17:35:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~175s 2024-01-15T17:35:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~175s 2024-01-15T17:35:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~175s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~175s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~175s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~175s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~181s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~181s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~175s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~175s 2024-01-15T17:35:12.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~181s 2024-01-15T17:35:12.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~181s 2024-01-15T17:35:12.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~175s 2024-01-15T17:35:12.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~181s 2024-01-15T17:35:14.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:14.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:17.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~180s 2024-01-15T17:35:17.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~186s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~186s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~180s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~180s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~186s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~186s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~180s 2024-01-15T17:35:17.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~186s 2024-01-15T17:35:17.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:20.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:20.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:20.604 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:20.604 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:35:20.604 INFO:tasks.ceph:Archiving crash dumps... 2024-01-15T17:35:20.607 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082/remote/smithi077/crash 2024-01-15T17:35:20.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:20.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:22.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~185s 2024-01-15T17:35:22.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~185s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~191s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~191s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~185s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~185s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~191s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~191s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~185s 2024-01-15T17:35:22.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~191s 2024-01-15T17:35:23.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~190s 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~190s 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~190s 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~190s 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~190s 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~190s 2024-01-15T17:35:27.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~190s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~190s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~190s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~197s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~197s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~190s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~190s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~197s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~197s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~190s 2024-01-15T17:35:27.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~197s 2024-01-15T17:35:32.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~195s 2024-01-15T17:35:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~202s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~202s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~195s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~195s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~202s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~202s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~195s 2024-01-15T17:35:32.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~202s 2024-01-15T17:35:32.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:32.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:35.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:37.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~200s 2024-01-15T17:35:37.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2024-01-15T17:35:37.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2024-01-15T17:35:37.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~207s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~207s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~200s 2024-01-15T17:35:37.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~200s 2024-01-15T17:35:37.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~207s 2024-01-15T17:35:37.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~207s 2024-01-15T17:35:37.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~200s 2024-01-15T17:35:37.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~207s 2024-01-15T17:35:42.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~205s 2024-01-15T17:35:42.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2024-01-15T17:35:42.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2024-01-15T17:35:42.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~205s 2024-01-15T17:35:42.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~212s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~212s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~205s 2024-01-15T17:35:42.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~212s 2024-01-15T17:35:42.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~212s 2024-01-15T17:35:42.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~205s 2024-01-15T17:35:42.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~212s 2024-01-15T17:35:47.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~210s 2024-01-15T17:35:47.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~217s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~217s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~210s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~210s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~217s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~217s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~210s 2024-01-15T17:35:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~217s 2024-01-15T17:35:47.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:47.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:50.812 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:52.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~215s 2024-01-15T17:35:52.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2024-01-15T17:35:52.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2024-01-15T17:35:52.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~222s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~222s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2024-01-15T17:35:52.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~215s 2024-01-15T17:35:52.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~222s 2024-01-15T17:35:52.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~222s 2024-01-15T17:35:52.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~215s 2024-01-15T17:35:52.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~222s 2024-01-15T17:35:53.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:53.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:53.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:53.884 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:35:53.884 INFO:tasks.ceph:Shutting down logrotate 2024-01-15T17:35:53.886 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 257, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 222, in end self.thread.get() File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 184, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:35:53.907 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-15T17:35:53.917 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:35:53.930 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-15T17:35:53.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:35:53.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:35:56.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:35:57.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2024-01-15T17:35:57.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2024-01-15T17:35:57.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2024-01-15T17:35:57.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~220s 2024-01-15T17:35:57.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~220s 2024-01-15T17:35:57.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~220s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~220s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~220s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~220s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~227s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~227s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~220s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~220s 2024-01-15T17:35:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~227s 2024-01-15T17:35:57.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~227s 2024-01-15T17:35:57.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~220s 2024-01-15T17:35:57.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~227s 2024-01-15T17:36:02.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~225s 2024-01-15T17:36:02.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~225s 2024-01-15T17:36:02.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~225s 2024-01-15T17:36:02.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~225s 2024-01-15T17:36:02.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2024-01-15T17:36:02.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~225s 2024-01-15T17:36:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~225s 2024-01-15T17:36:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~225s 2024-01-15T17:36:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~225s 2024-01-15T17:36:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~232s 2024-01-15T17:36:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~232s 2024-01-15T17:36:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~225s 2024-01-15T17:36:02.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~225s 2024-01-15T17:36:02.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~232s 2024-01-15T17:36:02.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~232s 2024-01-15T17:36:02.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~225s 2024-01-15T17:36:02.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~232s 2024-01-15T17:36:05.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:36:05.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:36:07.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~230s 2024-01-15T17:36:07.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~230s 2024-01-15T17:36:07.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~230s 2024-01-15T17:36:07.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~230s 2024-01-15T17:36:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~230s 2024-01-15T17:36:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~230s 2024-01-15T17:36:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~230s 2024-01-15T17:36:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~230s 2024-01-15T17:36:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~230s 2024-01-15T17:36:07.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~237s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~237s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~230s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~230s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~237s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~237s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~230s 2024-01-15T17:36:07.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~237s 2024-01-15T17:36:09.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:36:12.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~235s 2024-01-15T17:36:12.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~235s 2024-01-15T17:36:12.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~235s 2024-01-15T17:36:12.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~235s 2024-01-15T17:36:12.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~235s 2024-01-15T17:36:12.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~235s 2024-01-15T17:36:12.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~235s 2024-01-15T17:36:12.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~235s 2024-01-15T17:36:12.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~235s 2024-01-15T17:36:12.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~242s 2024-01-15T17:36:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~242s 2024-01-15T17:36:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~235s 2024-01-15T17:36:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~235s 2024-01-15T17:36:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~242s 2024-01-15T17:36:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~242s 2024-01-15T17:36:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~235s 2024-01-15T17:36:12.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~242s 2024-01-15T17:36:17.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~240s 2024-01-15T17:36:17.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~240s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~247s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~247s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~240s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~240s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~247s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~247s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~240s 2024-01-15T17:36:17.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~247s 2024-01-15T17:36:21.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:36:21.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:36:22.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~245s 2024-01-15T17:36:22.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~245s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~245s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~252s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~252s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~245s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~245s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~252s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~252s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~245s 2024-01-15T17:36:22.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~252s 2024-01-15T17:36:24.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:36:27.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:36:27.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:36:27.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:36:27.196 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:36:27.244 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:36:27.260 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-15T17:36:27.270 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-15T17:36:27.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:36:27.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:36:27.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2024-01-15T17:36:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~250s 2024-01-15T17:36:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~250s 2024-01-15T17:36:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~250s 2024-01-15T17:36:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~250s 2024-01-15T17:36:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~250s 2024-01-15T17:36:27.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~250s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~250s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~250s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~257s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~257s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~250s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~250s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~257s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~257s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~250s 2024-01-15T17:36:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~257s 2024-01-15T17:36:30.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:36:32.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2024-01-15T17:36:32.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~255s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~262s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~262s 2024-01-15T17:36:32.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~255s 2024-01-15T17:36:32.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~255s 2024-01-15T17:36:32.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~262s 2024-01-15T17:36:32.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~262s 2024-01-15T17:36:32.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~255s 2024-01-15T17:36:32.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~262s 2024-01-15T17:36:37.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2024-01-15T17:36:37.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2024-01-15T17:36:37.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~260s 2024-01-15T17:36:37.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~267s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~267s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~260s 2024-01-15T17:36:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~260s 2024-01-15T17:36:37.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~267s 2024-01-15T17:36:37.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~267s 2024-01-15T17:36:37.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~260s 2024-01-15T17:36:37.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~267s 2024-01-15T17:36:39.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:36:39.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:36:42.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:36:42.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2024-01-15T17:36:42.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-01-15T17:36:42.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~265s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~272s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~272s 2024-01-15T17:36:42.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~265s 2024-01-15T17:36:42.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~265s 2024-01-15T17:36:42.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~272s 2024-01-15T17:36:42.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~272s 2024-01-15T17:36:42.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~265s 2024-01-15T17:36:42.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~272s 2024-01-15T17:36:47.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2024-01-15T17:36:47.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-01-15T17:36:47.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2024-01-15T17:36:47.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~270s 2024-01-15T17:36:47.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~270s 2024-01-15T17:36:47.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~270s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~270s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~270s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~270s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~277s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~277s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~270s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~270s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~277s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~277s 2024-01-15T17:36:47.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~270s 2024-01-15T17:36:47.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~277s 2024-01-15T17:36:52.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~275s 2024-01-15T17:36:52.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~275s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~282s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~282s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~275s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~275s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~282s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~282s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~275s 2024-01-15T17:36:52.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~282s 2024-01-15T17:36:54.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:36:54.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:36:57.468 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:36:57.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~280s 2024-01-15T17:36:57.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~280s 2024-01-15T17:36:57.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~280s 2024-01-15T17:36:57.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~280s 2024-01-15T17:36:57.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~280s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~280s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~280s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~280s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~280s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~287s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~287s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~280s 2024-01-15T17:36:57.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~280s 2024-01-15T17:36:57.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~287s 2024-01-15T17:36:57.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~287s 2024-01-15T17:36:57.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~280s 2024-01-15T17:36:57.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~287s 2024-01-15T17:37:00.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:00.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:00.544 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:00.544 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:37:00.544 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:37:00.556 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-15T17:37:00.565 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-15T17:37:00.566 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-15T17:37:00.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:00.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:02.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~285s 2024-01-15T17:37:02.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~285s 2024-01-15T17:37:02.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~285s 2024-01-15T17:37:02.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~285s 2024-01-15T17:37:02.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~285s 2024-01-15T17:37:02.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~285s 2024-01-15T17:37:02.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~285s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~285s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~285s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~292s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~292s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~285s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~285s 2024-01-15T17:37:02.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~292s 2024-01-15T17:37:02.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~292s 2024-01-15T17:37:02.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~285s 2024-01-15T17:37:02.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~292s 2024-01-15T17:37:03.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:07.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~290s 2024-01-15T17:37:07.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~290s 2024-01-15T17:37:07.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~290s 2024-01-15T17:37:07.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~290s 2024-01-15T17:37:07.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~290s 2024-01-15T17:37:07.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~290s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~290s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~290s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~297s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~297s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~290s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~290s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~297s 2024-01-15T17:37:07.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~297s 2024-01-15T17:37:07.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~290s 2024-01-15T17:37:07.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~297s 2024-01-15T17:37:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~295s 2024-01-15T17:37:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~295s 2024-01-15T17:37:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~295s 2024-01-15T17:37:12.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~295s 2024-01-15T17:37:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~295s 2024-01-15T17:37:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~295s 2024-01-15T17:37:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~295s 2024-01-15T17:37:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~295s 2024-01-15T17:37:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~295s 2024-01-15T17:37:12.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~302s 2024-01-15T17:37:12.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~302s 2024-01-15T17:37:12.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~295s 2024-01-15T17:37:12.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~295s 2024-01-15T17:37:12.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~302s 2024-01-15T17:37:12.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~302s 2024-01-15T17:37:12.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~295s 2024-01-15T17:37:12.680 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~302s 2024-01-15T17:37:12.680 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-01-15T17:37:12.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:12.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:12.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:12.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:16.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:16.767 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:25.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:25.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:28.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:28.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:28.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:28.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:31.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:31.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:31.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:31.900 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:37:31.900 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:37:31.907 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-15T17:37:31.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-15T17:37:31.927 INFO:teuthology.task.internal:Duration was 11592.826216 seconds 2024-01-15T17:37:31.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-15T17:37:31.935 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-15T17:37:31.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:31.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:35.004 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:40.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:40.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:43.904 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:44.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:44.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:46.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:46.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:46.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:46.972 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:37:46.972 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:37:46.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:46.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:46.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:50.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:37:58.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:58.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:37:59.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:37:59.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:02.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:02.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:05.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:05.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:05.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:05.116 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:38:05.117 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:38:05.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-15T17:38:05.140 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-15T17:38:05.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:05.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:08.221 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:14.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:14.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:17.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:17.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:17.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:20.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:20.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:20.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:20.188 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:38:20.189 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:38:20.197 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:20.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:38:20.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:32.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:32.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:35.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:38.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:38.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:38.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:38.332 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:38:38.332 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:38:38.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-15T17:38:38.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:38.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:38.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:38:41.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:38:47.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:38:47.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:50.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:38:50.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:38:50.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:38:53.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:02.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:39:02.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:05.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:05.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:05.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:39:08.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:08.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:39:08.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:08.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:39:08.732 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:39:08.732 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi159 2024-01-15T17:39:08.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:39:08.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:11.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:11.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:11.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:11.644 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:39:11.645 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:39:11.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-15T17:39:11.669 INFO:teuthology.task.internal:Transferring archived files... 2024-01-15T17:39:11.670 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517082/remote/smithi077 2024-01-15T17:39:11.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:11.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:11.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:39:14.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:20.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:39:20.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:23.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:23.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:23.872 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:39:26.812 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:35.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:39:35.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:36.096 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:39:38.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:38.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:38.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:39.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-01-15T17:39:39.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:41.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:41.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:42.108 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-01-15T17:39:42.108 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:39:42.109 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi159 2024-01-15T17:39:44.925 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:44.926 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:39:44.927 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:39:44.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-15T17:39:44.953 INFO:teuthology.task.internal:Not uploading archives. 2024-01-15T17:39:44.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-15T17:39:44.968 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-15T17:39:44.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:44.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:39:47.996 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:39:57.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:39:57.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:40:00.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:40:12.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:40:12.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:40:12.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:40:15.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-15T17:40:15.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:40:18.236 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-15T17:40:18.236 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-15T17:40:18.237 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:40:18.250 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-15T17:40:18.263 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-15T17:40:18.273 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionError: Failed to reconnect to smithi077 2024-01-15T17:40:18.273 INFO:teuthology.nuke:Checking targets against current locks 2024-01-15T17:40:18.305 INFO:teuthology.nuke:Not nuking smithi077.front.sepia.ceph.com because description doesn't match: rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi != /home/teuthworker/archive/rishabh-2024-01-15_17:39:15-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517110 2024-01-15T17:40:18.329 INFO:teuthology.nuke:Not nuking smithi159.front.sepia.ceph.com because description doesn't match: rishabh-2024-01-15_14:18:41-fs:volumes-rishabh-clone-stats-testing-default-smithi != /home/teuthworker/archive/rishabh-2024-01-15_17:39:15-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517110 2024-01-15T17:40:18.330 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAm4BXul+XHUCskQiKefJarpQ9/RKYyRFJFCclOpgH06y89Jd2ZojDz9DQif+pbcTJGbWq9qELPAYP9m+tbmoHw= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSr5y6GfvbWAbi7CR/OsML6+UDb2zXubrvEUEnenhEBJeiHQSGr42UrAhK7RFT/ydM05oOJJLic2pNGBqryQ5g= 2024-01-15T17:40:18.338 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 11592.826215744019 failure_reason: 'Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9d008ac1c66c426986dcc1c80b6608cf status: fail success: false 2024-01-15T17:40:18.338 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T17:40:18.416 INFO:teuthology.run:FAIL